Data engineering
Part 6 of 6 · Approximate AggregationsProduction Sketch Ops — Serialization, Idempotent Merges, Bias & Monitoring
Sketches fail in production through bytes, retries, and silent bias — not through forgetting the paper. Version the payload, apply once by sketch_id, and watch coverage plus dual-read error.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
How you trust an approximate number
Prefer
Versioned bytes + exactly-once apply + dual-read
Compact binary with a schema envelope. At-least-once transport, exactly-once apply by sketch_id. Coverage and |p99_sketch − p99_exact_sample| are first-class signals.
- Nightly exact on 1% keys catches shape change.
- Dual-write old/new parameters before a read flip.
- Do not merge across tenants.
Alternative
JSON centroids and merge-on-every-retry
Bloated, unstable, and double-counted. Parameter changes mid-flight without dual-read look like “the p99 moved” when you moved k.
- Wrong-type deserialize is a silent incident.
- Dropped leaves look like a latency win.
- Unbounded sketch keys are cardinality — bound them.
Ops control plane
Trust is not the paper. Trust is apply-once plus monitors.
- 1
Leaf serializes v1
Native compact bytes plus type, lib version, k/compression. - 2
Transport at-least-once
Retries are expected. Ids must make them safe. - 3
Idempotent apply
sketch_id in an applied set with TTL ≥ window retention. - 4
Coverage + dual-read
leaves_present / expected, and |approx − exact_sample|. - 5
Dup without id check
Weight inflation, quantile shift, dedup_hit flat.
Overview
Sketches fail in production through bytes, retries, and silent bias — not through forgetting the paper. This lesson covers serialization, idempotent merges, bias sources, and monitoring so approximate aggregations stay trustworthy.
Topology (leaf → region → global) lives on merge pipelines. This page is the control plane.
You should be able to:
- Write the envelope fields.
- Diagnose double-merge vs coverage gap from symptoms.
- Roll a new k without a silent quantile jump.
Serialization
- Use library-native binary (DataSketches compact; T-Digest byte encoding) — not JSON of all centroids unless you are debugging.
- Version the payload:
sketch_type,lib_version, k / compression, endianness. - Compress on the wire (zstd) after
sketch.compact()— sketches are already lossy-compact. - Schema registry / protobuf wrapper:
{window, key, sketch_id, bytes}.
JSON centroids are bloated and unstable across lang ports. Debug dumps are fine; the merger path is compact + version.
Idempotent merges
sketch_id= hash(tenant, window, leaf_id, epoch, content_hash)- Merger stores applied ids (TTL ≥ window retention)
- At-least-once transport + exactly-once apply
- Re-merge after crash: safe if ids deduped; unsafe if “just merge bytes again”
Key contents are not only the leaf name. Two payloads from the same leaf in one window must not collide if the content changed; the same bytes must collide.
Bias and correctness hazards
| Hazard | What you see | Fix |
|---|---|---|
| Double-count on retry | Weight up, quantiles shift, dedup_hit flat | sketch_id apply-once |
| Under-count on dropped leaves | Optimistic p99, coverage gap | Alert coverage % |
| Clock / window mis-alignment | Regions disagree on which window | Align windowing; document lateness |
| Parameter drift | k or compression changed mid-flight | Dual-write, then flip read |
| Wrong type | KLL bytes into a T-Digest parser | Version + type in the envelope |
| Calibration drift | Sketch vs exact sample diverges after traffic shape change | Dual-read time series, nightly 1% exact |
Monitoring (what to alert)
merge_success/merge_dedup_hitratiossketch_bytes_p99, merge latency- coverage = leaves_present / leaves_expected per window
- dual_read_error =
|p99_sketch − p99_exact_sample|on canary keys parameter_versionmismatch count
Limit sketch keys the way you limit metric labels — metric cardinality. Ops still needs correct aggregation behind an SLI panel — histogram SLIs — without redefining SLOs here.
Streaming vs batch ops
- Streaming: flush interval vs memory; backpressure when the merger lags.
- Batch: partition sketch files in object store; deterministic tree merge for reproducibility.
- Nightly exact recalculation on 1% keys to catch bias regressions.
Security / multi-tenant
- Do not merge across tenants.
- Size-limit sketch bytes to avoid decompression bombs.
- ACLs on the quantile API same as raw metric access.
Architecture (emit → apply → observe)
Flow
- 1
1 Leaf serialize v1
- next2 Transport at-least-once
- 2
2 Transport at-least-once
- next3 Idempotent apply by sketch id
- relatedFail: dup without id check
- 3
3 Idempotent apply by sketch id
- next4 Coverage plus dual-read
- 4
4 Coverage plus dual-read
- next5 Alert on bias or dedup anomalies
- 5
5 Alert on bias or dedup anomalies
- 6
Fail: dup without id check
Lesson map
Production Sketch Ops — Serialization, Idempotent Merges, Bias & Monitoring
Sketches fail in production through bytes, retries, and silent bias — not through forgetting the paper. Version the payload, apply once by sketch_id, and watch coverage plus dual-read error.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB l["1 Leaf serialize v1"] t["2 Transport at-least-once"] i["3 Idempotent apply by sketch id"] c["4 Coverage plus dual-read"] l -->|1 Leaf serialize v1| t t -->|2 Transport at-least-once to 3| i i -->|3 Idempotent apply by sketch id| c
Sandbox: idempotent apply + bias check (Python)
Educational envelope. Production uses library-native bytes and a real SHA-256; the playground hashes payload with the stdlib.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Same idea (TypeScript)
No Node crypto in the playground. A tiny string hash stands in for SHA-256.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Pitfalls
p99 drops 40% at 14:07. merge_success is up. dedup_hit is unchanged. Weight for window 14:00 is ~2×. Coverage is 100%. What happened? What if instead coverage is 60% and weight is down — what do you tell the on-call?
Interview Q&A
Why not JSON centroids?
Answer
Bloated and unstable across languages. Use compact binary plus a versioned envelope. JSON is a debug dump.
What goes in the idempotency key?
Answer
Tenant, window, leaf, payload hash — not only the leaf name. Optionally epoch. TTL on the applied set covers the window retention.
Symptom of double-merge?
Answer
Total weight up, quantiles shift, dedup_hit flat. Coverage usually still looks fine.
How do you detect bias?
Answer
Dual-read exact on sampled keys; track error as a time series. Nightly exact on 1% keys after traffic-shape changes.
How do you roll out a new k?
Answer
Dual-write old and new sketches; compare dual-read; flip read. Never change k mid-window on a single blob.
Multi-tenant risk?
Answer
Cross-merge leaks distributions. Isolate by tenant in the envelope and in the store. ACLs on the quantile API match raw metrics.
Link to cardinality docs?
Answer
Limit sketch keys like metric labels. Unbounded keys explode the merger the same way they explode Prometheus. Cross-link metric cardinality.
Link to histogram SLIs?
Answer
Ops still needs correct aggregation behind the SLI panel. Do not redefine SLOs here — histogram vs average.
Streaming vs batch ops?
Answer
Streaming: flush vs memory vs merger backpressure. Batch: object-store partition files and a deterministic tree for replay. Hybrid: stream plus nightly sample-exact.
What if deserialize hits the wrong type?
Answer
Envelope sketch_type plus lib version. KLL bytes into a T-Digest parser is a pageable mismatch count, not a “weird p99.”