DevOps & platform
Observability
When production breaks, metrics, logs, and traces together tell you what, where, and why — a single log tail rarely can.
✗ Problem
Production is a black box
A user reports "it's slow." You grep scattered logs across a dozen services and still can't answer what broke, where, or why.
# three unrelated log files, no shared request id:
$ grep "error" api.log auth.log payment.log
# thousands of lines… no timing, no correlationNo timeline, no latency breakdown, no way to link one user's request across
services — just noise.
✓ How it works
↓
↓
Three pillars, one story
Each pillar answers a different question about the same request, then feeds dashboards and alerts.
# Prometheus metric
http_request_duration_seconds{p99} 0.42
// OpenTelemetry span
span("payment.charge", ms=900)App
emits telemetry
📈 Metrics
rate · errors · p99
📝 Logs
timestamped events
🔗 Traces
per-span timing
Dashboards / Alerts
Grafana · SLOs
✓ See it live
Incident: "it's slow" at 14:02
Step through the three pillars to pinpoint the root cause of one slow request.
WHAT + WHEN — p99 latency spikes at 14:02.
WHERE — the trace shows 900ms spent inside Payment.
API Gateway
50ms
Auth Service
30ms
Payment Service
900ms
Notify Service
20ms
WHY — the payment service log reveals the cause.
14:02:01 INFO payment-svc received charge id=8841
14:02:01 WARN payment-svc db pool waiting (5/5 in use)
14:02:02 ERROR payment-svc DB pool exhausted after 900ms
✓ Takeaway
Correlate all three
- Metrics = what/when — cheap, aggregated, always-on numbers.
- Traces = where — which service/span ate the time.
- Logs = why — the detailed event that explains the failure.
- Instrument once with OpenTelemetry; export to any backend.
- Define SLIs/SLOs and alert on user-facing symptoms, not raw CPU%.
- Careful: high-cardinality labels (user IDs, raw URLs) blow up metrics cost.
Pairs with resilience patterns;
eBPF can capture this telemetry with near-zero overhead.
Back to all topics →