It's 2 AM. An alert fires. Error rate on the checkout service spiked to 12% starting eight minutes ago. Your on-call engineer is awake, staring at a Grafana dashboard that shows the spike — and nothing else.
They check CPU: normal. Memory: normal. Database connections: can't tell, that metric wasn't set up. They search logs: unstructured text, no correlation IDs, impossible to filter to just the failing requests. They look at recent deployments: nothing in the last 6 hours. They're going to be awake for a while.
This is a monitoring problem masquerading as an operations problem. The system is telling the engineer that something is wrong. It is not helping them understand why — and that gap is the difference between a 20-minute incident and a 3-hour one.
Monitoring: Knowing That Something Is Wrong
Monitoring, in the traditional sense, is the practice of collecting predefined metrics and alerting when they cross thresholds. CPU utilization. Memory usage. Error rate. Request latency. HTTP status codes. These are things you know to measure in advance, expressed as time-series data.
Monitoring is necessary. You absolutely should know when your error rate spikes or your latency degrades. The problem is that monitoring only tells you what you already knew to look for. It answers questions you anticipated.
Production systems fail in ways nobody anticipated. The database connection pool exhausts because a new query pattern was introduced that holds connections longer. A background job creates a memory leak under a specific combination of inputs that only appears during a particular traffic pattern. A third-party API begins returning malformed responses that only affect users in a specific region. None of these failures are captured by the standard monitoring stack because nobody thought to write those alerts in advance.
Observability: Understanding Why
Observability — borrowed from control systems theory — refers to the ability to infer the internal state of a system from its external outputs. An observable system is one where you can ask arbitrary questions about what it's doing without having to predict those questions in advance.
In practice, observability in software systems is built on three types of telemetry data, commonly called the three pillars:
Logs
Logs have existed since the beginning of computing. The difference between logs that support observability and logs that don't is structure. Unstructured log lines like ERROR: payment failed are searchable only if you know what string to search for. Structured logs — JSON with consistent field names for request IDs, user identifiers, service names, durations, and error codes — are filterable, correlatable, and aggregatable.
The practice that makes structured logs useful is consistent correlation IDs. Every request that enters your system should be assigned a unique trace ID that propagates through every service it touches. When an error occurs, you can filter logs across all services for that single trace ID and reconstruct exactly what happened, in sequence, across your entire system.
Metrics
Metrics are aggregated numerical measurements over time. Request rates, latency percentiles, queue depths, cache hit rates. They're excellent for dashboards and for alerting because they're cheap to store and query at scale.
The evolution in modern metrics practice is moving from infrastructure-centric metrics (CPU, memory, disk) toward RED metrics (Rate, Errors, Duration) and USE metrics (Utilization, Saturation, Errors) applied at the service level. These metrics are closer to the user experience and more directly actionable when something goes wrong.
Service-level objectives (SLOs) formalize this: instead of alerting on CPU > 80%, you alert on error budget burn rate — a measure of how quickly you're consuming the allowable error budget for a service. This aligns on-call alerting with the things that actually matter to users.
Traces
Distributed traces are the pillar that monitoring systems typically lack entirely, and they're what make the difference in complex microservices environments.
A trace represents the full journey of a single request through your system. It captures every service the request touched, how long each step took, what data was passed between services, and where errors occurred. A flame graph view of a trace makes it immediately obvious that your checkout request spent 720ms of its 847ms total waiting for a database connection — not in your application code, not in your network, but specifically in the PostgreSQL connection pool.
Monitoring tells you the what. Observability tells you the why. You need both, but the industry spent two decades building the former and is only now catching up on the latter.
The OpenTelemetry Standard
The practical challenge with observability until recently was that logs, metrics, and traces each had their own instrumentation libraries, their own wire formats, and their own vendor lock-in. OpenTelemetry changes this.
OpenTelemetry is a CNCF project that provides a single, vendor-neutral SDK for instrumenting applications to emit all three types of telemetry. You instrument once against the OpenTelemetry API, and the backend — Jaeger, Honeycomb, Datadog, Grafana Tempo, Google Cloud Trace — is a configuration choice rather than a code change.
For new services, adding OpenTelemetry auto-instrumentation is straightforward. For popular frameworks (Express, Django, Spring Boot, Go net/http), the auto-instrumentation libraries handle trace propagation and basic span creation without code changes to your business logic. Custom spans for business-critical paths — payment processing, inventory checks, user authentication — require a handful of additional lines but yield enormous diagnostic value.
Choosing a Backend
The observability backend landscape has consolidated considerably. For most teams starting out, the decision comes down to:
Grafana Stack (OSS) — Loki for logs, Prometheus for metrics, Tempo for traces, unified in Grafana dashboards. Excellent if you want full control and have the operational capacity to run it. The Grafana Agent or OpenTelemetry Collector handles data collection. Total cost is infrastructure + your time.
Honeycomb — purpose-built for high-cardinality observability with an exceptional query interface for arbitrary exploratory analysis. More expensive than self-hosted but saves significant operational overhead. Best in class for teams that prioritize incident investigation speed over cost.
Datadog — comprehensive, mature, integrates everything. Also expensive, with pricing that compounds in ways that surprise teams at scale. Worth it for organizations that need the breadth and are willing to pay for it.
Google Cloud's operations suite (formerly Stackdriver) — if you're on GCP, Cloud Logging, Cloud Monitoring, and Cloud Trace integrate seamlessly with managed services like GKE and Cloud Run. Not best-in-class for distributed tracing but competent and cost-effective if you're already in the GCP ecosystem.
Starting Without Ripping Everything Out
The most common mistake is treating observability as a big-bang migration. Teams spend months evaluating backends, debating instrumentation strategies, and then run out of momentum before anything ships.
A more productive approach is to start with the highest-value surface and expand from there:
- Structured logging first. Add JSON formatting to your log output and a correlation ID that propagates through your request context. This is a low-cost change with immediate diagnostic value.
- Add OpenTelemetry auto-instrumentation to your most critical services. Pick your payment service, your API gateway, your main backend — wherever incidents have the biggest blast radius. Get traces flowing to a backend.
- Define SLOs for those services. Error budget alerting replaces a wall of threshold-based alerts with a smaller number of higher-signal ones.
- Expand coverage incrementally. Each new service that gets instrumented increases the value of the traces you already have, because the full call graph becomes more complete.
What Changes When You Get There
Teams with mature observability handle incidents differently. The on-call engineer who gets paged at 2 AM doesn't start by checking CPU and memory. They open a trace for one of the failing requests, see immediately that the payment service is waiting 340ms for a database connection, check the database connection pool metrics, find it saturated, and have a fix deployed in under 20 minutes.
The difference isn't skill. It's information. The observable system gave the engineer everything they needed to diagnose the root cause from a single trace. The monitored system gave them a number that was wrong and left them to figure out the rest.
You cannot build software you can't see into. Monitoring is a start. Observability is what you actually need.