Observability Triad — Metrics, Logs & Distributed Tracing

Overview

The observability triad is metrics (aggregates over time), logs (discrete events), and traces (causal request trees across services). Events and exemplars glue them together. Senior engineers pick the right pillar first, then correlate with shared identifiers — not dump everything into one unstructured firehose.

The same production incident answers different questions depending on which pillar you open:

You should be able to:

1. Name which pillar wins for budget burn, per-request forensics, and critical-path latency.
2. Sketch detect → scope → detail → decide (metrics → trace → logs → page / ticket / freeze).
3. Define an exemplar and why every log line carries trace_id.
4. Contrast RED vs USE and say when each is the first dashboard.
5. List the three classic anti-patterns interviewers love.

flowchart TB
  app["App"]
  prom["Prometheus"]
  tempo["Traces"]
  loki["Logs"]
  app -->|histogram plus| prom
  app -->|spans same| tempo
  app -->|structured log| loki
  prom -->|open exemplar| tempo
  tempo -->|filter by| loki

1. 1

   error_rate / latency / saturation → page or heatmap

   Detect on metrics

   Cheap if labels are bounded. Perfect for SLOs, alerting, and capacity. You know that something burned — not which request.

2. 2

   exemplar / alert link → span waterfall

   Scope on a trace

   The histogram bucket points at a trace id. Open the waterfall. That is the hop that added 800 ms — not "payments is slow on average."

3. ?

   same trace_id → page / ticket / freeze

   Detail on structured logs

   Filter logs by that id. Then decide: remediate now, file debt, or freeze features because the budget is gone.

Pillar deep-dives

Metrics — when they beat the alternatives

Pros: cheap cardinality if labels are bounded; perfect for SLOs, alerting, capacity; histograms plus exemplars link to traces.

Cons: lose the per-request story; high-cardinality labels explode memory; averages lie — use histograms.

Anti-pattern: http_requests_total{user_id=...} → metric explosion. Design labels on metric cardinality. Latency SLIs are threshold ratios, not means — that argument lives on histogram vs average.

Use metrics first for: error-budget burn, "do we need more pods?", "did this deploy move p99?", saturation. Those questions are SLI / SLO / budget and burn-rate alerts.

Logs — when they beat the alternatives

Pros: rich context, cheap to add fields; forensic detail; audit trails.

Cons: unstructured text is unqueryable at scale; logging every debug line in prod is a DDoS on yourself; without trace_id you cannot join to traces.

Anti-pattern: log.info(f"got {body}") with PII and no schema.

Use logs first when you already have a request id / user id / audit obligation. Do not start a Sev-1 by grepping "error" across 40 services.

Traces — when they beat the alternatives

Pros: causal graph across services; critical path; tail-latency debugging.

Cons: storage cost; head-based sampling misses rare failures unless combined with tail sampling; broken context propagation = orphan spans.

Anti-pattern: sample 100% of prod traffic with no budget; or sample 0.1% with no error bias.

W3C traceparent, parent-based flags, and head vs tail are a full lesson: trace context & sampling. The September 15 cluster's trace sampling strategies is the keep/drop policy deep dive — use both pages, do not paste them together.

Exemplars and structured correlation

An exemplar is a metric sample that points at a trace id (Prometheus exemplars / OpenTelemetry). Alert fires on p99 → click exemplar → open that slow trace → jump to logs with the same trace_id.

Without exemplars you correlate by timestamp across clock-skewed hosts. That is swivel-chair observability.

The pairing of buckets ↔ traces ↔ RED/USE dashboards is exemplars, RED & USE. This hub's job is the workflow: detect, scope, detail.

RED vs USE

Checkout 500s on idle CPUs is a RED incident. Disk at 100% with a rising wait queue is a USE incident that becomes RED in minutes. Alert on user SLIs; keep USE on investigation dashboards, not as the primary page. Full pairing: exemplars / RED / USE.

Correlation demo (in-memory)

I/O: fake checkout hops in → counters, JSON logs, and a span tree out. No OpenTelemetry SDK — this is the data model.

Press Run. Snippets must be self-contained — no network, files, or native modules.

Press Run. Snippets must be self-contained — no network, files, or native modules.

You should see 200 and 502 counters, JSON lines that share a trace_id with the span tree, and an exemplar pointing at the last request's id. That is the join key — not the wall clock.

Pros / cons summary

Interview Q&A

Pitfalls

1. Alerting on infrastructure metrics while user SLIs are green or red unnoticed.
2. Trace every request with no sampling budget.
3. Metric labels from unconstrained user input.
4. Logs without schema / trace_id / severity.
5. Three tools, zero correlation — swivel-chair observability.
6. Starting a Sev-1 in the log search box.
7. Averages as a latency SLO (use histograms; see the SLOs cluster).
8. Mesh-only traces with no app spans — network hops without business logic.

Go Deeper

• OpenTelemetry — Signals
• Google SRE Book — Monitoring Distributed Systems
• Prometheus — Histograms and summaries
• Grafana — Exemplars
• Grafana — The RED method
• Brendan Gregg — USE method
• YouTube: Observability: the present and future (Charity Majors)
• Companion cluster: SLOs, error budgets & tracing

Cheat sheet

Metrics = aggregates (SLOs, burn, capacity)
Logs    = events with context (forensics, audit)
Traces  = causal span trees (critical path)

Detect on metrics → exemplar → trace → logs(trace_id) → decide

RED = Rate, Errors, Duration     (services)
USE = Utilization, Saturation, Errors  (resources)

Anti-patterns: unstructured logs, user_id labels, 100% tracing