SLI Design Patterns

Overview

An SLI (Service Level Indicator) is a quantitative measure of reliability from the consumer’s perspective. The SRE shape is a ratio over a window:

SLI = good events / valid events     # 0–1, report as a percent

The SLO is the target for that ratio (99.9% over 28 or 30 days). The SLA is the external contract. This lesson is the indicator: what you count, what you drop, and which pattern matches the failure users feel. The hub covers the contract; burn alerts consume this ratio.

If you cannot name the event and the predicate for good, you are not ready to pick 99.9%. Interviewers stop you there.

Why the ratio wins over a vibe metric

1. 1

   vanity metric → green board, dead checkout

   Ping /healthz and call it availability

   Probes are not users. Mixing them into valid events inflates the ratio. CPU "looks fine" is the same class of mistake.

2. 2

   Winner: user-centric good / valid, one class at a time

   Name the event (checkout HTTP, homepage paint, pipeline record). Drop invalid. Count good. Bucket CRITICAL vs HIGH_SLOW. Latency uses a threshold ratio, not an average.

3. ?

   Then attach a window, SLO, and burn policy

   Same ratio feeds 14.4× pages and 1× tickets. If the SLI is wrong, every alert downstream is theater. Journey SLIs need traces; freshness needs timestamps — pick the pattern that matches the pain.

Good events vs valid events

Two counters, always:

• Valid — attempts that should count toward the promise. Successful or failed user-facing calls. This is the denominator.
• Good — the subset that met the promise (2xx/3xx, or latency under the threshold, or data younger than T). This is the numerator.

Request success ratio = good / valid over the measurement window (fixed 5-minute scrape for alerting; rolling ~4-week window for the SLO).

Invalid is not “failed.” Invalid is “this row should not be in the fraction.” Typical drops:

• Load-balancer health checks and synthetics you do not want in the product SLO (or keep them in a separate probe SLI).
• Auth bot 401/403 if product policy says those are client mistakes, not your unavailability.
• Internal scrapes, admin, and dark-launch traffic.
• Duplicate retries of the same user attempt — see below.

flowchart TB
  a["Incoming request"]
  b["Valid user attempt?"]
  d["Drop from denominator"]
  c["Start timer"]
  a -->|Incoming request to Valid user attempt?| b
  b -->|No| d
  b -->|Yes| c

Step notes: 1 validate (denominator), 2 time the work, 3 apply the success predicate, 4 observe latency whether good or bad, 5 export. The histogram is not the availability SLI; it feeds a latency-threshold SLI.

Availability, latency, freshness, quality

Availability (request-driven APIs)

Good = responses that are successful by your contract — often non-5xx, sometimes “business-OK” (HTTP 200 with paid: false may still be a failed charge). Valid = user-facing attempts that reached a complete response or timed out (timeouts are bad, not invalid).

Do not invent a unique SLO per endpoint. Bucket:

• CRITICAL — login, checkout, pay (tightest).
• HIGH_FAST — interactive reads.
• HIGH_SLOW — exports, reports (looser latency, still high availability).
• LOW / NO_SLO — polls, debug, dark launches.

Latency (threshold ratio, not a percentile nested in a percentile)

“p99 under 300 ms” is a useful dashboard. It is a clumsy budget. The SLI that composes with availability is:

good = requests with duration ≤ T
SLI  = good / valid
SLO  = that ratio ≥ 99%   (example)

Put an exact histogram bucket on T. Missing le="0.3" biases the ratio. Averages hide the tail — full worked numbers on histogram vs average.

Freshness (pipelines, caches, replicas)

Users care that the data they read is young enough, not that the job “ran.” Emit age at the point of consumption (now − record_ts). Good = age under threshold (for example 5 s for a session cache, 15 min for a warehouse tile). Valid = reads (or records) that should have been fresh.

You need trustworthy timestamps: NTP or the producer’s event time, not a clock that drifted 30 s on one AZ. OpenTelemetry trace IDs do not replace event time; they correlate the read to the writer.

Quality / correctness

Graceful degradation is still a worse user experience: stale catalog, generic avatar, approximate search. Quality SLI = undegraded responses / valid. Correctness = schema or business-rule pass rate (golden synthetic, independent checker). Expensive to compute; use on money paths and ML scores, not on every static asset.

User-journey (composite) SLIs

Checkout is not “payments 99.9%” and “inventory 99.9%” multiplied in your head. A journey SLI is one end-to-end event: the user started checkout and received a paid confirmation within T, with no manual retry. That needs a shared trace id (W3C traceparent) so you do not double-count hops. Per-request SLIs stay for component contracts with dependencies.

A hybrid “good” means succeeded and fast. That is stricter than two separate SLOs; say so, or you will double-punish the budget. Many teams keep two SLIs (availability, latency) on the same valid stream and two budgets.

Where you measure

Separate SLI specification (“homepage becomes interactive under 100 ms”) from SLI implementation (edge http_request_duration under 100 ms). Document the gap: server-side misses DNS and CDN failures. Start at the LB, iterate toward the user.

Measurement windows: short (1–5 min) for burn alerts; long (~28–30 days, integer weeks so weekend count stays stable) for the SLO. Do not report a 30-day SLO off a 5-minute scrape without recording rules.

Worked ratios (in-memory)

No scrape, no HTTP. A tiny request log in memory: each row is one attempt with flags for retry, probe, latency, payload age, and degradation. The functions below are the I/O contract: in → classified events, out → printed SLI ratios.

Press Run. Snippets must be self-contained — no network, files, or native modules.

Read the output: the probe and the successful retry make a naive success ratio look healthier than the user journey. The 2.5 s call fails the latency SLI while still “available.” The stale degraded 200 fails freshness and quality. That is the whole point of several SLIs on the same valid stream — one number cannot name every failure mode.

Instrument the same split in production with two counters (requests_valid, requests_good) plus a histogram whose buckets include the latency threshold. OTel and Prometheus are exporters; they do not choose the predicate for you.

Pipelines, coverage, durability

Request APIs are not the only consumers. The workbook’s other SLI families show up in interviews as soon as you mention Kafka, a warehouse, or object storage.

Coverage. Expected records vs successfully processed records. A consumer that “is up” at 99.9% while dropping 8% of messages is a coverage miss, not an availability miss. Valid = records the contract said you would handle (offset high-water, files in the landing bucket). Good = those that made it to the sink with a checksum.

Durability. Readable records / written records. Users often care about today’s objects, not the ten-year archive: say which population you count. A 99.999% durability SLO on “all objects ever” can hide a bad day of new writes.

Correctness vs freshness on the same pipeline. A job can be on time and wrong (schema drift) or late and right (catch-up). Two SLIs, two budgets. Do not OR them into one “pipeline healthy” boolean — that boolean cannot feed a burn rate.

Worked availability still applies: 99.9% over 30 days is 43.2 minutes of equivalent full outage, or 0.1% of events. If checkout sees 3,000,000 attempts in the window, the budget is 3,000 failed attempts. Pick the unit that matches the SLI (time for “are we up,” events for “did this request work”). The hub computes both; this page decides which events exist.

Instrumentation without a scrape loop

A wrapper around a handler is enough conceptually:

1. Decide valid (user attempt, not probe, not extra retry).
2. Start a monotonic timer.
3. Run the work.
4. Apply the good predicate (status contract, and/or duration ≤ T, and/or age ≤ T).
5. Observe the histogram always (failed calls have latency too).
6. Export via OTel or Prometheus — the backend is not the SLI.

Reuse the wrapper across endpoints; put class (CRITICAL / HIGH_SLOW) on labels, not a unique SLO name per URL. Externalize T and the class map so a latency SLO change is config, not a binary. High-frequency debug endpoints can sample debug attributes; the SLI counters themselves stay on 100% of valid events — sampling traces is a different knob (trace sampling).

Client-side retries plus idempotency keys: the user journey is still one attempt. Count the journey once. The retry storm belongs on a dependency RED panel, not as extra good/valid pairs.

Interview Q&A

Pitfalls

1. Counting retries as good — one painful checkout becomes a success after attempt 4.
2. Health-check inflation — /healthz 200s in valid while checkout is dead.
3. Average latency as an SLI — tail disappears; cannot aggregate means of means.
4. Window mismatch — alerting on 30 days of scrape noise, or reporting SLO off 5 minutes.
5. Clock skew on freshness — threshold of 5 s with 8 s NTP drift.
6. Hard-coded thresholds in binaries — T and class maps should be config, not a redeploy.
7. Over-instrumenting every URL — high-cardinality labels (user id) explode series; sample hot debug paths, keep SLIs on low-cardinality class/route.
8. Mixed units — milliseconds in one service, seconds in another; standardize on seconds for OTel histograms.
9. One SLO for every endpoint — unmaintainable; bucket classes.
10. Treating a dependency outage as “not our SLI” — users do not care whose queue failed; product SLIs stay on user impact.

Go Deeper

• Google SRE Book — Monitoring Distributed Systems
• Google SRE Workbook — Implementing SLOs
• Google SRE Book — Data Integrity (freshness / correctness cousins)
• OpenTelemetry — Metrics
• Prometheus — Histograms and summaries
• Next in cluster: Multi-window burn alerts

Cheat sheet

SLI = good / valid          (user attempt, not probe, not extra retry)
Availability: success by contract / valid
Latency:     duration ≤ T / valid     (histogram bucket on T)
Freshness:   age ≤ T / valid          (event time, not span duration)
Quality:     undegraded / valid
Journey:     end-to-end good / valid  (needs trace id)

Then: SLO target, budget = 1 − SLO, multi-window burn