Your metrics dashboard is green. Your error rate is 0.1%. And yet your users are filing support tickets saying the checkout flow "feels slow." You open Datadog, click through five dashboards, and still can't tell whether the slowness is in your API gateway, the inventory service, the payment processor call, or the database query that runs inside it.
This is the gap that OpenTelemetry fills — and after years of competing standards, vendor lock-in, and instrumentation fragmentation, it's finally the unambiguous right choice. OTel 1.0 traces and metrics are stable, the logs spec is production-ready, and every major observability backend supports it. There's no longer a good reason to reach for a vendor SDK first.
This post walks through what it actually takes to instrument a production service with OpenTelemetry end to end — collector config, sampling strategy, context propagation, and the pitfalls that will bite you if you skip them.
OpenTelemetry is an instrumentation standard and SDK collection. It defines how you generate, collect, and export traces, metrics, and logs from your services. It does not store or visualize that data — that's still the job of backends like Grafana Tempo, Jaeger, Honeycomb, or Datadog.
The important distinction: OTel gives you vendor-neutral instrumentation. You instrument once with the OTel SDK, and you can swap backends without touching your application code. If you instrumented with the Datadog SDK directly, migrating to Honeycomb means re-instrumenting everything.
The core components you'll work with:
Before writing a line of code, make a conscious choice here — most teams get this wrong by defaulting entirely to one approach.
Auto-instrumentation uses agents or byte-code manipulation to instrument common libraries automatically. For Java, the OTel Java agent instruments JDBC, Spring, Kafka, gRPC, and dozens more with a single -javaagent flag. For Node.js, @opentelemetry/auto-instrumentations-node wraps Express, HTTP, and database clients automatically. For Python, opentelemetry-instrument does the same.
This gets you 80% of the way there with zero application changes. Start here.
# Node.js auto-instrumentation
npm install @opentelemetry/auto-instrumentations-node @opentelemetry/sdk-node
# Add to your entry point before anything else
node --require @opentelemetry/auto-instrumentations-node/register app.js
# Python
pip install opentelemetry-distro
opentelemetry-bootstrap --action=install
opentelemetry-instrument python app.py
Manual instrumentation with the OTel SDK is for the 20% that matters most — your business logic, critical paths, and any custom operations that auto-instrumentation can't see inside. Adding spans around your actual domain operations is what makes traces readable and useful.
// Go — manual span creation
tracer := otel.Tracer("checkout-service")
func ProcessOrder(ctx context.Context, orderID string) error {
ctx, span := tracer.Start(ctx, "ProcessOrder",
trace.WithAttributes(attribute.String("order.id", orderID)),
)
defer span.End()
if err := validateInventory(ctx, orderID); err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, err.Error())
return err
}
// ...
}
The key discipline: always propagate context. Pass ctx through every function call on a hot path. If you fire a goroutine or async task without forwarding the context, you'll break the trace and lose the span relationship. This is the single most common mistake teams make with manual instrumentation.
Teams sometimes skip the collector and export directly from the application SDK to the backend. Don't. The collector handles:
A minimal collector config for a Kubernetes deployment:
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 5s
send_batch_size: 1000
memory_limiter:
check_interval: 1s
limit_mib: 512
resource:
attributes:
- key: environment
value: production
action: upsert
exporters:
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
prometheusremotewrite:
endpoint: http://prometheus:9090/api/v1/write
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch, resource]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch, resource]
exporters: [prometheusremotewrite]
Two processors you must include: memory_limiter (prevents OOM kills under load) and batch (dramatically reduces export overhead). Running the collector without both in production will cause you pain.
Tracing every request at scale is expensive. A service handling 10,000 RPS generating complete traces will overwhelm your storage budget fast. You need a sampling strategy.
Head-based sampling decides at trace creation whether to record the trace. Simple and cheap, but it means you're as likely to drop a slow trace as a fast one — exactly the traces you need most.
Tail-based sampling buffers complete traces and makes the keep/drop decision after the fact, based on the full trace data. Keep all error traces, keep slow traces, sample healthy-fast traces at 5%. This is what you actually want in production.
Configure tail sampling in the collector:
processors:
tail_sampling:
decision_wait: 10s
num_traces: 50000
policies:
- name: errors-policy
type: status_code
status_code: { status_codes: [ERROR] }
- name: slow-traces-policy
type: latency
latency: { threshold_ms: 500 }
- name: probabilistic-policy
type: probabilistic
probabilistic: { sampling_percentage: 5 }
The decision_wait value (10s here) is the window to buffer a complete trace before deciding. Set it too low and you'll make sampling decisions on incomplete traces. Set it too high and your collector memory requirements balloon.
Distributed tracing only works if trace context crosses service boundaries correctly. OTel uses the W3C TraceContext standard by default — a traceparent header that carries the trace ID and parent span ID.
For HTTP services, auto-instrumentation handles this automatically. For anything else — gRPC, Kafka messages, async queues, background jobs — you're responsible for propagating the context manually.
# Injecting context into a Kafka message header
from opentelemetry import propagate
headers = {}
propagate.inject(headers)
producer.send("orders", value=payload, headers=list(headers.items()))
# Extracting context when consuming
context = propagate.extract(dict(message.headers))
with tracer.start_as_current_span("consume-order", context=context):
process(message)
If you're seeing traces that stop at a service boundary with no downstream spans, missing context propagation is almost always why.
The fastest way to make your metrics backend unusable — and expensive — is high-cardinality labels. Every unique combination of label values creates a new time series. Labels that look innocent at first become monsters at scale.
Never use these as metric labels:
/orders/12345/items)Safe label values:
/orders/{id}/items)The OTel SDK won't stop you from shooting yourself in the foot here. This is a discipline you have to enforce in code review and collector config. The collector's filter and transform processors can scrub high-cardinality attributes before they reach your backend.
Don't try to instrument everything at once. A practical rollout:
The whole process typically takes two to four weeks for a team of three moving deliberately. The collector config and sampling policies will need tuning for the first month as you understand your traffic patterns.
Forgetting to set resource attributes. Every span should carry service.name, service.version, and deployment.environment. Without these, you can't filter traces by service in your backend. Set them in the collector's resource processor so you don't rely on every team remembering to do it in code.
Exporting directly from SDK in production. Always use the collector. Direct export under load drops data and adds latency to your application threads.
Ignoring the collector's memory limits. Under a traffic spike, a collector without memory_limiter will OOM and drop everything. Set the limit to 80% of the container's memory allocation.
Treating spans like logs. Spans represent operations with start and end times. Don't attach large blobs of data as span attributes. Keep attribute values short and structured.
OpenTelemetry has reached the point where there's no good argument for vendor-specific instrumentation on new services. The ecosystem is mature, the backends all speak OTLP, and the instrumentation investment is portable.
If you're starting a new service or modernizing an existing one and want observability that actually answers questions at 2 AM — let's talk.