There's a question that almost every on-call engineer has asked at 2am: "The alert fired, I can see something is broken — but why?"
That question is what separates monitoring from observability.
Monitoring is the practice of collecting and alerting on known failure modes. You instrument your system, define what "bad" looks like, and get paged when those thresholds are crossed. It answers: "Is something wrong?"
Observability is the property of a system that lets you understand its internal state from external outputs — without having to predict in advance what questions you'll need to ask. It answers: "Why is something wrong, and what exactly is happening?"
Monitoring tells you your error rate spiked. Observability lets you trace which requests are failing, what's different about them, and where in the call chain the failure originates.
Logs are the oldest observability signal. A well-structured log line tells you exactly what happened at a specific moment for a specific request.
The difference between useful logs and useless logs is almost entirely about structure. Plain-text log lines like "Payment failed" are nearly worthless in production. Structured JSON logs with request IDs, user IDs, error codes, and durations are invaluable.
{
"level": "error",
"timestamp": "2026-05-07T14:22:04Z",
"request_id": "a3f9b2c1",
"service": "payment-api",
"handler": "ProcessPayment",
"error": "upstream_timeout",
"upstream": "stripe-api",
"duration_ms": 5003,
"user_id": "usr_1234"
}
This log tells a story. The payment failed because Stripe timed out after 5 seconds. You have the request ID to correlate with traces, the user ID for customer support, and the handler name to direct the investigation.
Metrics are aggregated measurements over time. They're good for trends, capacity planning, and alerting on known conditions. They're bad for debugging individual requests — by definition, they throw away the per-request detail that makes debugging possible.
The key metrics to instrument for any service:
These four (popularized as RED and USE methodologies) give you a baseline picture of service health.
Traces are the newest and most powerful pillar for debugging distributed systems. A trace follows a single request through every service it touches — showing timing, errors, and context at each step.
When a request is slow, a trace tells you exactly where the time was spent: 2ms in service A, 800ms in service B, 5ms in service C. The 800ms in service B is where you look.
Without traces, debugging distributed systems is educated guessing. With traces, it's an investigation with evidence.
OpenTelemetry (OTel) is the open standard for instrumenting applications to emit logs, metrics, and traces. It's vendor-neutral, has SDKs for every major language, and lets you switch backend observability platforms without re-instrumenting your code.
The basic setup:
The beauty of OTel is that your instrumentation code is the same regardless of whether you're sending to Grafana, Datadog, Honeycomb, or GCP Cloud Trace.
Grafana stack (Loki + Prometheus + Tempo): Open-source, self-hosted, full control. Good choice if you have the operational capacity to run it. Significant setup investment.
Honeycomb: Purpose-built for observability. Excellent trace UI and query interface. More expensive, but engineers who use it love it. Good for teams prioritizing fast debugging.
Datadog: Comprehensive commercial platform. Does everything, integrates with everything. Costs grow fast at scale.
GCP Cloud Operations (formerly Stackdriver): Native integration with GCP services. Lower friction if you're already on GCP. Trace and log correlation works well.
For teams starting out: if you're on GCP, start with Cloud Operations — the zero-friction integration is worth it. If you're cloud-agnostic, the Grafana stack gives you full control without vendor lock-in.
You don't have to instrument everything at once. A practical order:
The systems that are hardest to operate are the ones that are hardest to understand. Observability isn't about buying a tool — it's about building systems that tell you what they're doing.
The investment pays off every time you cut an incident resolution time from two hours to twenty minutes.
We help engineering teams instrument their services and build out observability pipelines. Get in touch to talk through your setup.