OpenTelemetry in Production: Instrumenting Services Without Losing Your Mind

Your metrics dashboard is green. Your error rate is 0.1%. And yet your users are filing support tickets saying the checkout flow "feels slow." You open Datadog, click through five dashboards, and still can't tell whether the slowness is in your API gateway, the inventory service, the payment processor call, or the database query that runs inside it.

This is the gap that OpenTelemetry fills — and after years of competing standards, vendor lock-in, and instrumentation fragmentation, it's finally the unambiguous right choice. OTel 1.0 traces and metrics are stable, the logs spec is production-ready, and every major observability backend supports it. There's no longer a good reason to reach for a vendor SDK first.

This post walks through what it actually takes to instrument a production service with OpenTelemetry end to end — collector config, sampling strategy, context propagation, and the pitfalls that will bite you if you skip them.

What OpenTelemetry Actually Is (and Isn't)

OpenTelemetry is an instrumentation standard and SDK collection. It defines how you generate, collect, and export traces, metrics, and logs from your services. It does not store or visualize that data — that's still the job of backends like Grafana Tempo, Jaeger, Honeycomb, or Datadog.

The important distinction: OTel gives you vendor-neutral instrumentation. You instrument once with the OTel SDK, and you can swap backends without touching your application code. If you instrumented with the Datadog SDK directly, migrating to Honeycomb means re-instrumenting everything.

The core components you'll work with:

OTel SDK — the in-process library your application uses to create spans, record metrics, and emit logs
OTel Collector — a standalone agent/gateway that receives telemetry, processes it, and exports to backends
OTLP — the wire protocol (gRPC or HTTP/JSON) that everything speaks

Auto-Instrumentation vs Manual SDK: Pick the Right Tool

Before writing a line of code, make a conscious choice here — most teams get this wrong by defaulting entirely to one approach.

Auto-instrumentation uses agents or byte-code manipulation to instrument common libraries automatically. For Java, the OTel Java agent instruments JDBC, Spring, Kafka, gRPC, and dozens more with a single -javaagent flag. For Node.js, @opentelemetry/auto-instrumentations-node wraps Express, HTTP, and database clients automatically. For Python, opentelemetry-instrument does the same.

This gets you 80% of the way there with zero application changes. Start here.

# Node.js auto-instrumentation
npm install @opentelemetry/auto-instrumentations-node @opentelemetry/sdk-node

# Add to your entry point before anything else
node --require @opentelemetry/auto-instrumentations-node/register app.js

# Python
pip install opentelemetry-distro
opentelemetry-bootstrap --action=install
opentelemetry-instrument python app.py

Manual instrumentation with the OTel SDK is for the 20% that matters most — your business logic, critical paths, and any custom operations that auto-instrumentation can't see inside. Adding spans around your actual domain operations is what makes traces readable and useful.

// Go — manual span creation
tracer := otel.Tracer("checkout-service")

func ProcessOrder(ctx context.Context, orderID string) error {
    ctx, span := tracer.Start(ctx, "ProcessOrder",
        trace.WithAttributes(attribute.String("order.id", orderID)),
    )
    defer span.End()

    if err := validateInventory(ctx, orderID); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }
    // ...
}

The key discipline: always propagate context. Pass ctx through every function call on a hot path. If you fire a goroutine or async task without forwarding the context, you'll break the trace and lose the span relationship. This is the single most common mistake teams make with manual instrumentation.

The OTel Collector: Non-Negotiable in Production

Teams sometimes skip the collector and export directly from the application SDK to the backend. Don't. The collector handles:

Buffering and retry — your backend goes down, the collector queues telemetry
Tail-based sampling — you can't make head-sampling decisions at the application layer for traces you haven't finished yet
Fan-out — send to multiple backends simultaneously during a migration
Attribute processing — strip PII, add environment tags, normalize service names before data leaves your cluster

A minimal collector config for a Kubernetes deployment:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  resource:
    attributes:
      - key: environment
        value: production
        action: upsert

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true
  prometheusremotewrite:
    endpoint: http://prometheus:9090/api/v1/write

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch, resource]
      exporters: [prometheusremotewrite]

Two processors you must include: memory_limiter (prevents OOM kills under load) and batch (dramatically reduces export overhead). Running the collector without both in production will cause you pain.

Sampling: The Decision You Can't Skip

Tracing every request at scale is expensive. A service handling 10,000 RPS generating complete traces will overwhelm your storage budget fast. You need a sampling strategy.

Head-based sampling decides at trace creation whether to record the trace. Simple and cheap, but it means you're as likely to drop a slow trace as a fast one — exactly the traces you need most.

Tail-based sampling buffers complete traces and makes the keep/drop decision after the fact, based on the full trace data. Keep all error traces, keep slow traces, sample healthy-fast traces at 5%. This is what you actually want in production.

Configure tail sampling in the collector:

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces-policy
        type: latency
        latency: { threshold_ms: 500 }
      - name: probabilistic-policy
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

The decision_wait value (10s here) is the window to buffer a complete trace before deciding. Set it too low and you'll make sampling decisions on incomplete traces. Set it too high and your collector memory requirements balloon.

Context Propagation Across Service Boundaries

Distributed tracing only works if trace context crosses service boundaries correctly. OTel uses the W3C TraceContext standard by default — a traceparent header that carries the trace ID and parent span ID.

For HTTP services, auto-instrumentation handles this automatically. For anything else — gRPC, Kafka messages, async queues, background jobs — you're responsible for propagating the context manually.

# Injecting context into a Kafka message header
from opentelemetry import propagate

headers = {}
propagate.inject(headers)
producer.send("orders", value=payload, headers=list(headers.items()))

# Extracting context when consuming
context = propagate.extract(dict(message.headers))
with tracer.start_as_current_span("consume-order", context=context):
    process(message)

If you're seeing traces that stop at a service boundary with no downstream spans, missing context propagation is almost always why.

The Cardinality Trap

The fastest way to make your metrics backend unusable — and expensive — is high-cardinality labels. Every unique combination of label values creates a new time series. Labels that look innocent at first become monsters at scale.

Never use these as metric labels:

User IDs, request IDs, order IDs
IP addresses
Full URL paths with IDs embedded (e.g., /orders/12345/items)

Safe label values:

HTTP method, status code
Service name, version
Route template (e.g., /orders/{id}/items)
Environment, region

The OTel SDK won't stop you from shooting yourself in the foot here. This is a discipline you have to enforce in code review and collector config. The collector's filter and transform processors can scrub high-cardinality attributes before they reach your backend.

Putting It Together: Rollout Strategy

Don't try to instrument everything at once. A practical rollout:

Deploy the collector first — get the infrastructure running before touching application code
Auto-instrument your highest-traffic services — immediate visibility with minimal risk
Add manual spans to your critical paths — checkout, authentication, data pipelines
Wire up context propagation for async boundaries — Kafka, SQS, background workers
Configure tail sampling once you have baseline traffic — tune thresholds against real data
Add metrics and logs last — traces give you the most immediate debugging value; metrics and logs fill in the gaps

The whole process typically takes two to four weeks for a team of three moving deliberately. The collector config and sampling policies will need tuning for the first month as you understand your traffic patterns.

Common Pitfalls

Forgetting to set resource attributes. Every span should carry service.name, service.version, and deployment.environment. Without these, you can't filter traces by service in your backend. Set them in the collector's resource processor so you don't rely on every team remembering to do it in code.

Exporting directly from SDK in production. Always use the collector. Direct export under load drops data and adds latency to your application threads.

Ignoring the collector's memory limits. Under a traffic spike, a collector without memory_limiter will OOM and drop everything. Set the limit to 80% of the container's memory allocation.

Treating spans like logs. Spans represent operations with start and end times. Don't attach large blobs of data as span attributes. Keep attribute values short and structured.

OpenTelemetry has reached the point where there's no good argument for vendor-specific instrumentation on new services. The ecosystem is mature, the backends all speak OTLP, and the instrumentation investment is portable.

If you're starting a new service or modernizing an existing one and want observability that actually answers questions at 2 AM — let's talk.