Observability vs. Monitoring: Why It Matters for Production Systems

There's a question that almost every on-call engineer has asked at 2am: "The alert fired, I can see something is broken — but why?"

That question is what separates monitoring from observability.

Monitoring is the practice of collecting and alerting on known failure modes. You instrument your system, define what "bad" looks like, and get paged when those thresholds are crossed. It answers: "Is something wrong?"

Observability is the property of a system that lets you understand its internal state from external outputs — without having to predict in advance what questions you'll need to ask. It answers: "Why is something wrong, and what exactly is happening?"

Monitoring tells you your error rate spiked. Observability lets you trace which requests are failing, what's different about them, and where in the call chain the failure originates.

The three pillars

Logs

Logs are the oldest observability signal. A well-structured log line tells you exactly what happened at a specific moment for a specific request.

The difference between useful logs and useless logs is almost entirely about structure. Plain-text log lines like "Payment failed" are nearly worthless in production. Structured JSON logs with request IDs, user IDs, error codes, and durations are invaluable.

{
  "level": "error",
  "timestamp": "2026-05-07T14:22:04Z",
  "request_id": "a3f9b2c1",
  "service": "payment-api",
  "handler": "ProcessPayment",
  "error": "upstream_timeout",
  "upstream": "stripe-api",
  "duration_ms": 5003,
  "user_id": "usr_1234"
}

This log tells a story. The payment failed because Stripe timed out after 5 seconds. You have the request ID to correlate with traces, the user ID for customer support, and the handler name to direct the investigation.

Metrics

Metrics are aggregated measurements over time. They're good for trends, capacity planning, and alerting on known conditions. They're bad for debugging individual requests — by definition, they throw away the per-request detail that makes debugging possible.

The key metrics to instrument for any service:

• Request rate — how much traffic are you handling?
• Error rate — what percentage is failing?
• Latency distribution — p50, p95, p99 (don't just track averages)
• Saturation — how close are you to capacity?

These four (popularized as RED and USE methodologies) give you a baseline picture of service health.

Traces

Traces are the newest and most powerful pillar for debugging distributed systems. A trace follows a single request through every service it touches — showing timing, errors, and context at each step.

When a request is slow, a trace tells you exactly where the time was spent: 2ms in service A, 800ms in service B, 5ms in service C. The 800ms in service B is where you look.

Without traces, debugging distributed systems is educated guessing. With traces, it's an investigation with evidence.

OpenTelemetry: the standard

OpenTelemetry (OTel) is the open standard for instrumenting applications to emit logs, metrics, and traces. It's vendor-neutral, has SDKs for every major language, and lets you switch backend observability platforms without re-instrumenting your code.

The basic setup:

1. Add the OTel SDK to your service
2. Configure an exporter (sends telemetry to your backend)
3. Add instrumentation (automatic for most HTTP frameworks; manual for business logic)

The beauty of OTel is that your instrumentation code is the same regardless of whether you're sending to Grafana, Datadog, Honeycomb, or GCP Cloud Trace.

Choosing a backend

Grafana stack (Loki + Prometheus + Tempo): Open-source, self-hosted, full control. Good choice if you have the operational capacity to run it. Significant setup investment.

Honeycomb: Purpose-built for observability. Excellent trace UI and query interface. More expensive, but engineers who use it love it. Good for teams prioritizing fast debugging.

Datadog: Comprehensive commercial platform. Does everything, integrates with everything. Costs grow fast at scale.

GCP Cloud Operations (formerly Stackdriver): Native integration with GCP services. Lower friction if you're already on GCP. Trace and log correlation works well.

For teams starting out: if you're on GCP, start with Cloud Operations — the zero-friction integration is worth it. If you're cloud-agnostic, the Grafana stack gives you full control without vendor lock-in.

Getting started incrementally

You don't have to instrument everything at once. A practical order:

1. Structured logging everywhere — this has the lowest effort and highest immediate value
2. Request IDs — ensure every request gets a unique ID that propagates through your services and appears in every log line
3. Basic metrics (RED) — request rate, error rate, latency
4. Alerting on error rate and latency — alert on symptoms, not causes
5. Distributed tracing — start with your critical path (user-facing request flow)

---

The systems that are hardest to operate are the ones that are hardest to understand. Observability isn't about buying a tool — it's about building systems that tell you what they're doing.

The investment pays off every time you cut an incident resolution time from two hours to twenty minutes.

We help engineering teams instrument their services and build out observability pipelines. Get in touch to talk through your setup.