Multi-Window Burn Alerts

Overview

A multi-window burn alert watches how fast the error budget is being spent over several overlapping windows (classically 1 hour, 6 hours, 3 days). It fires only when the burn in a long window and a short window both exceed a threshold. Severity scales with how quickly the 30-day (or 28-day) budget would vanish.

This is the SRE workbook recipe from Alerting on SLOs. It is not a PromQL hobby: it is how you get paged for a fast outage and ticketed for a slow leak, without training the team to ignore 10-minute threshold flaps.

The SLI that feeds the burn must already be honest — SLI design patterns. Latency burns use a threshold-ratio, not an average — histograms.

Why multi-window AND-gates win

1. 1

   naive threshold → fatigue or missed leaks

   Page when error rate ≥ SLO for 10 minutes

   A quiet hour with one failure looks like a 1000× burn. A week-long 0.15% leak never trips 10 minutes. This is the alert people learn to ignore.

2. 2

   Winner: burn rate × window, long AND short

   burn = error_rate / (1 − SLO). Classic page: 14.4× on 1 h AND 5 m (~2% of a 30-day budget). Second page: 6× on 6 h AND 30 m (~5%). Ticket: 1× on 3 d AND 6 h (~10%).

3. ?

   When it fires, jump with traces — do not raise a third threshold

   Exemplars pin a trace to the histogram bucket. Sampling decides whether that trace exists. Those are later lessons. Here, get the page right first.

Error budget and burn rate

SLI          = good / valid
SLO          = target                 # 0.999 for 99.9%
error_budget = 1 − SLO                # 0.001
error_rate   = 1 − SLI                # bad / valid
burn_rate    = error_rate / error_budget

Burn 1× means: if this error rate continues, you exhaust the budget at the end of the SLO window. Burn 14.4× means you would exhaust it 14.4 times faster — in window / 14.4.

For 99.9% over 30 days:

• Window = 30 × 24 × 60 = 43,200 minutes
• Budget time = 0.001 × 43,200 = 43.2 minutes of equivalent full outage
• If the SLI is event-based, budget = 0.1% of requests, not wall clock. Same math: replace minutes with events.

Quantify incidents as percent of budget burned, not as “sev-2 vibes.”

The classic workbook table (99.9%, ~30-day window)

Page when both windows exceed the burn. Short window ≈ 1/12 of the long window so the alert resets quickly after burning stops (the 1 h condition would otherwise keep paging for an hour).

Worked companions:

• 6× for 6 h: 5% of 43.2 min = 2.16 min equivalent. 2.16 / 360 = 0.006 error rate → 0.006 / 0.001 = 6.
• 1× for 3 d: 3/30 of the window at budget pace → 10% of monthly budget.

For 99.95% or 99.99%, recompute; do not copy 14.4 blindly. Extreme 99.999% monthly ≈ 26 seconds of full outage — you cannot page your way out; you need canaries and blast-radius limits.

AND-gate, not three independent sirens

flowchart TB
  d1["1h burn"]
  g["1h ≥ 14.4× AND 5m ≥ 14.4×"]
  f1["3d burn"]
  i["3d ≥ 1× AND 6h ≥ 1×"]
  d1 -->|1h burn to 1h ≥ 14.4× AND 5m ≥ 14.4×| g
  f1 -->|3d burn to 3d ≥ 1× AND 6h ≥ 1×| i

Why AND. A 30-second full outage:

• 5 m window: 30/300 = 0.10 error rate → burn 100× (trips short).
• 1 h window: 30/3600 ≈ 0.0083 → burn 8.3× (under 14.4).
• AND fails → no page. Correct: 30 s is ~1.2% of a 43.2 min budget, not a 2% hour-burn.

A 5-minute full outage: 1 h error rate 5/60 ≈ 0.083 → burn 83×; 5 m burn is huge. AND fires. You just spent 5 / 43.2 ≈ 12% of monthly budget — worth a page.

After the incident stops, the 5 m window heals in minutes; the 1 h window would keep you up. The short leg is the reset.

Suppress overlapping severities: if 14.4× is paging, do not also open the 1× ticket for the same burn. Route page → human; ticket → next business day.

Conceptual rule (sketch only — not scraped here):

page_fast =
  rate_1h > 14.4 * 0.001
  AND
  rate_5m > 14.4 * 0.001

page_mid =
  rate_6h > 6 * 0.001
  AND
  rate_30m > 6 * 0.001

ticket =
  rate_3d > 1 * 0.001
  AND
  rate_6h > 1 * 0.001

Recording rules pre-aggregate ratio_rate1h and friends so Alertmanager is not computing 3-day rates on every eval. On-the-fly queries are fresher and more expensive at high cardinality.

Low traffic and other teeth

One failed request in a 10-request hour → error rate 0.1 → burn 100×. The 14.4× page will fire even though you spent almost no monthly budget. Mitigations: synthetic traffic so the denominator is never tiny; combine related services; a minimum-request guard; or a lower SLO / longer window for that class.

Without a written error-budget policy (freeze, reliability focus, escalation), the page is just noise. When budget is exhausted: stop risky ships. When in budget and users are happy: ship. When met but users hate you: tighten the SLI, not the alert math.

Worked numbers (in-memory)

I/O: constants and synthetic error rates in, printed burns and AND-gate decisions out. No Prometheus.

Press Run. Snippets must be self-contained — no network, files, or native modules.

Expected shape: budget minutes 43.2; 14.4× hour consumes 2%; 6× / 6 h consumes 5%; 1× / 3 d consumes 10%; 30 s blip does not page; 5 min outage does; the 0.15% leak tickets but does not 14.4× page; 1/10 requests looks like a 100× fire.

Windows, 28 days, and other SLOs

Integer weeks (28 days) keep weekend count stable in a rolling window. 28-day budget at 99.9% is 0.001 × 28 × 24 × 60 ≈ 40.3 minutes. Recompute 14.4: 2% of 40.3 min in one hour is 0.02 × (28 × 24) = 13.44×, not 14.4. Teams that say “14.4” on a 28-day SLO are approximating; say the approximation out loud or derive the constant from target_fraction × window_hours.

Other useful derivations (always: burn = (budget_fraction_you_want_to_spend) × (window / alert_long_window)):

99.99% paging at 144× means a ~25-second full outage in an hour trips the “2% of budget” page. That is how tight 99.99% is — not a PromQL trick.

Inhibit / suppress. Alertmanager should inhibit the 1× ticket while a 14.4× or 6× page is firing for the same SLI, and inhibit 6× while 14.4× is already paging if they represent the same incident. Otherwise on-call gets three issues for one outage.

Recording rules (sketch): emit slo:burn_rate:ratio_rate1h, …5m, …6h, …30m, …3d as error_ratio / budget. Alert on those time series. Evaluating a 3-day rate() on every rule group without recording is how you learn about Prometheus CPU.

Latency-threshold SLIs use the same table: error_rate is the fraction slower than T (or failed). Do not invent a second 10-minute p99 alert and call it the SLO page.

Policy teeth. A page without freeze rules is a dashboard with sound. When 14.4× fires, the playbook is: halt risky deploys, find the burn with exemplars, restore the SLI, then spend remaining budget on purpose. When the 1× ticket fires, you still have ~90% of the window — schedule reliability work this week, do not wake someone at 3am.

Interview Q&A

Pitfalls

1. Alerting on error rate ≥ (1 − SLO) for 10 minutes — floods pages, misses leaks.
2. No AND-gate — every 5-minute spike pages; on-call dies.
3. Copied 14.4× onto a 99.99% SLO — the constant is for ~2% of a 30-day 99.9% budget; recompute.
4. Low-traffic paging — one fail at night looks like 1000×; add floors or synthetics.
5. Overlapping page + ticket — suppress the weaker severity for the same burn.
6. Thresholds too aggressive — paging on 2× / 5 m trains people to ignore the app.
7. Wrong SLI feeding the burn — retries and /healthz in valid; fix the indicator first.
8. No budget policy — you paged, then shipped the launch anyway.
9. 99.999% heroics — 26 seconds of outage exhaust the month; need progressive delivery, not a tighter PromQL.

Go Deeper

• Google SRE Workbook — Alerting on SLOs (the 14.4× recipe)
• Google SRE Workbook — Implementing SLOs
• Honeycomb — SLOs get budget rate alerts
• Grafana Cloud — SLO and burn-rate alerting
• Previous: SLI design patterns · Next: Histogram vs average latency

Cheat sheet

budget = 1 − SLO
burn   = error_rate / budget
99.9% / 30d budget ≈ 43.2 min

Page  14.4×  for 1h AND 5m    (~2% budget)
Page  6×     for 6h AND 30m   (~5% budget)
Ticket 1×    for 3d AND 6h    (~10% budget)

Do not page on “error rate ≥ SLO for 10 minutes”