Kubernetes Control Plane - API Server Request Path, etcd, Watches, Informers, Optimistic Concurrency & CRDs

Overview

The control plane is a REST API in front of a replicated key-value store, with a change feed. The API server is the only component that talks to etcd. It authenticates and authorizes each request, runs admission plugins that can rewrite or reject it, validates it against a schema, and stores it with a monotonically increasing resourceVersion. Every other component, including the scheduler and all controllers, is just a client that lists objects once and then watches for changes, keeping a local cache. Concurrency control is optimistic: a write that was computed from a stale read fails with 409 Conflict instead of silently overwriting someone else's change.

Get this layer right and the rest of Kubernetes stops being magic. Get it wrong and you see the classic incidents: a webhook outage that blocks every deploy, an etcd database that hits its quota, controllers that hammer the API server with full LISTs, or two automations that keep undoing each other.

The API server request path

flowchart TB
  r["Step 1: HTTPS request arrives with a client cert, bearer token or OIDC token"]
  n["Step 2: authentication maps the request to a user and groups"]
  f["Step 3: API Priority and Fairness picks a flow and priority level from that identity"]
  z["Step 4: authorization (Node, RBAC, webhook) checks verb, resource, namespace"]
  ma["Step 5: mutating admission: built-in plugins, MutatingAdmissionPolicy, mutating webhooks in series"]
  sv["Step 6: schema validation against the OpenAPI schema or CRD structural schema"]
  va["Step 7: validating admission: built-in plugins, ValidatingAdmissionPolicy, validating webhooks in parallel"]
  et["Step 8: etcd transaction compares the expected resourceVersion and writes"]
  wc["Step 9: watch cache receives the event and fans it out to watchers"]
  e1["Failure path: 401 Unauthorized"]
  e2["Failure path: 403 Forbidden"]
  e3["Failure path: every matching create or update fails cluster-wide"]
  e4["Failure path: 409 Conflict, client must re-read and retry"]
  r -->|continues| n
  n -->|continues| f
  f -->|continues| z
  z -->|continues| ma
  ma -->|continues| sv
  sv -->|continues| va
  va -->|continues| et
  et -->|continues| wc
  n -->|continues| e1
  z -->|continues| e2
  ma -->|continues| e3
  et -->|continues| e4

Facts worth remembering, all from the Kubernetes docs:

• Admission only sees writes. Reads (get, list, watch) bypass admission, so you cannot use admission to hide data; that is RBAC's job.
• Mutating webhooks run in series; validating webhooks run in parallel. Mutations happen before schema validation, so a mutation cannot produce an invalid object that slips through.
• Side effects need reconciliation. An admission plugin cannot know whether a later plugin will reject the request, so anything it changes elsewhere (quota counters, external records) needs a cleanup loop.
• The defaults matter. In Kubernetes 1.37 the default admission plugins include MutatingAdmissionPolicy, ValidatingAdmissionPolicy, both webhook plugins, PodSecurity, ResourceQuota, DefaultStorageClass and StorageObjectInUseProtection.

In-process CEL policies vs webhooks. A ValidatingAdmissionPolicy runs inside the API server, so it adds no network hop and cannot be "down". A webhook can call external systems and run arbitrary code, but it is a remote dependency on your critical write path. Prefer CEL policies for simple rules, and give webhooks tight timeouts, narrow rules and namespaceSelectors, and a deliberate failurePolicy.

Flow control. APF divides the API server's concurrency (by default --max-requests-inflight 400 plus --max-mutating-requests-inflight 200 are summed into the total when APF is on) into priority levels, and queues requests fairly per flow inside each level. The point is isolation: one controller stuck in a LIST loop gets throttled with 429s while node heartbeats and leader election keep flowing. Clients must honor 429 and Retry-After.

etcd: the store behind it

etcd is a Raft-replicated key-value store with MVCC: every write creates a new revision of the whole keyspace, and old revisions are kept until compacted. The API server maps each object to a key and uses the etcd revision of the last write as the object's resourceVersion. If you know Raft from the Raft series, the consequences follow directly.

Why etcd is the bottleneck. Every component's writes funnel through one Raft leader, every object lives in one keyspace, and every watcher depends on its revision stream. Kubernetes' published scale envelope (no more than 5,000 nodes, 150,000 pods, 300,000 containers, 110 pods per node) is largely a statement about what the API server and etcd can keep up with. The usual ways teams hurt it: storing megabytes in ConfigMaps or Secrets, creating thousands of short-lived custom resources or Events, and controllers that LIST the world on every loop.

Watches, resourceVersion and optimistic concurrency

The API server keeps an in-memory watch cache per resource type. Watches and most list requests are served from it rather than etcd. In Kubernetes 1.37, even "most recent" consistent reads can be served from the watch cache (with etcd 3.4.31+ or 3.5.13+), using etcd progress notifications to prove the cache is fresh.

Writes use the same number for optimistic concurrency. A PUT that carries a resourceVersion succeeds only if the object has not changed since; otherwise 409 Conflict. This is compare-and-swap, the same idea as in CAS and optimistic coordination, implemented with an etcd transaction.

"""Simulation: etcd-style revisions, optimistic concurrency and watch history.

Models three documented behaviours:
  * every write bumps one global revision; an object's resourceVersion is the
    revision of its last write
  * an update carrying a stale resourceVersion gets 409 Conflict
  * watches can resume only inside the retained history window; after
    compaction an old resourceVersion gets 410 Gone and the client must relist
"""
class Conflict(Exception): pass
class Gone(Exception): pass

class ToyEtcd:
    def __init__(self):
        self.rev = 0
        self.kv = {}          # key -> (value, mod_revision)
        self.history = []     # (revision, key, value)
        self.compacted = 0

    def put(self, key, value, expect_rv=None):
        cur = self.kv.get(key)
        if expect_rv is not None and cur and cur[1] != expect_rv:
            raise Conflict(f"409 Conflict: {key} is at rv {cur[1]}, you sent {expect_rv}")
        self.rev += 1                       # a single txn: compare then put
        self.kv[key] = (value, self.rev)
        self.history.append((self.rev, key, value))
        return self.rev

    def get(self, key):
        return self.kv[key]

    def compact(self, upto):
        self.history = [h for h in self.history if h[0] > upto]
        self.compacted = upto

    def watch_from(self, rv):
        if rv < self.compacted:
            raise Gone(f"410 Gone: rv {rv} older than compacted rv {self.compacted}")
        return [h for h in self.history if h[0] > rv]

db = ToyEtcd()
db.put("deploy/checkout", {"replicas": 3, "image": "v1"})

# Two writers read the same version...
hpa_view, hpa_rv = db.get("deploy/checkout")
ci_view, ci_rv = db.get("deploy/checkout")
print("both read rv", hpa_rv)

# ...the HPA scales first
db.put("deploy/checkout", {**hpa_view, "replicas": 6}, expect_rv=hpa_rv)
print("HPA wrote replicas=6 -> rv", db.get("deploy/checkout")[1])

# CI tries a full replace built from its stale read (PUT with resourceVersion)
try:
    db.put("deploy/checkout", {**ci_view, "image": "v2"}, expect_rv=ci_rv)
except Conflict as e:
    print("CI:", e)
    # RetryOnConflict pattern: re-read, re-apply the intent, write again
    fresh, rv = db.get("deploy/checkout")
    db.put("deploy/checkout", {**fresh, "image": "v2"}, expect_rv=rv)
    print("CI retried on rv", rv, "->", db.get("deploy/checkout"))

# Without the precondition, last writer wins and the HPA's scale-up is lost
lost = ToyEtcd(); lost.put("d", {"replicas": 3, "image": "v1"})
stale, _ = lost.get("d")
lost.put("d", {**stale, "replicas": 6})
lost.put("d", {**stale, "image": "v2"})          # blind write from stale copy
print("blind writes result:", lost.get("d")[0], "<- replicas=6 silently lost")

# Watches: resume inside the window, or relist after compaction
for i in range(5):
    db.put(f"pod/p{i}", {"phase": "Running"})
print("\ncurrent revision", db.rev)
print("watch from rv 4 ->", [r for r, _, _ in db.watch_from(4)])
db.compact(6)             # kube-apiserver asks etcd to compact on an interval (default 5m)
try:
    db.watch_from(4)
except Gone as e:
    print("watch from rv 4 ->", e)
    snapshot_rv = db.rev  # relist: read everything at a current revision
    print("relist at rv", snapshot_rv, "then watch from there ->", db.watch_from(snapshot_rv))

Output:

both read rv 1
HPA wrote replicas=6 -> rv 2
CI: 409 Conflict: deploy/checkout is at rv 2, you sent 1
CI retried on rv 2 -> ({'replicas': 6, 'image': 'v2'}, 3)
blind writes result: {'replicas': 3, 'image': 'v2'} <- replicas=6 silently lost

current revision 8
watch from rv 4 -> [5, 6, 7, 8]
watch from rv 4 -> 410 Gone: rv 4 older than compacted rv 6
relist at rv 8 then watch from there -> []

Two lessons from that run. First, the blind write lost the HPA's scale-up with no error at all; the 409 is a feature. Second, a watcher that falls behind the compaction point must relist, which is expensive on big clusters. BOOKMARK events exist so idle watchers can advance their resourceVersion and stay inside the window, and streaming lists (sendInitialEvents=true, beta and enabled by default in current releases) let a client get the initial state as a watch stream instead of one giant LIST response.

Server-side apply: optimistic concurrency per field

Whole-object CAS is coarse: the HPA changing replicas should not conflict with CI changing image. Server-side apply (SSA) tracks a field manager for every field in metadata.managedFields. Applying a field that another manager owns with a different value is a conflict that you either resolve or deliberately --force-conflicts.

Informers: how controllers read without melting the API server

A controller built with client-go does not poll. It uses an informer: a Reflector does one LIST and then one long-running WATCH, pushes changes through a DeltaFIFO into a thread-safe Indexer (the local cache), and calls event handlers. Handlers do not do the work; they put the object's key (namespace/name) on a rate-limited work queue. Workers pop keys, read the latest object from the cache, and reconcile. Because the queue holds each key at most once, ten updates to one object while it waits produce one reconcile, against the newest state.

// Simulation: why controllers use informers (list+watch into a local cache)
// and a de-duplicating work queue instead of polling the API server.
type Pod = { name: string; rv: number; phase: string };

let apiReads = 0;                        // requests that hit the API server
const server = new Map<string, Pod>();
for (let i = 0; i < 200; i++) server.set(`p${i}`, { name: `p${i}`, rv: i + 1, phase: "Running" });

// Generate a burst: 1000 status updates spread over just 20 pods (hot keys)
const events: Pod[] = [];
let rv = 1000;
for (let i = 0; i < 1000; i++) {
  const name = `p${i % 20}`;
  const pod = { name, rv: ++rv, phase: i % 7 === 0 ? "Pending" : "Running" };
  server.set(name, pod);
  events.push(pod);
}

// Approach A: naive controller does a GET of every pod for every event it hears about
function naive(): number {
  apiReads = 0;
  for (const _ev of events) { for (const _k of server.keys()) apiReads++; }  // "list everything to be safe"
  return apiReads;
}

// Approach B: informer = 1 LIST + 1 WATCH stream feeding a cache; handlers enqueue KEYS
class WorkQueue {
  private queue: string[] = [];
  private dirty = new Set<string>();       // a key waits in the queue at most once
  add(k: string) { if (!this.dirty.has(k)) { this.dirty.add(k); this.queue.push(k); } }
  get(): string | undefined { const k = this.queue.shift(); if (k) this.dirty.delete(k); return k; }
  get len() { return this.queue.length; }
}
function informer(): { reads: number; reconciles: number; staleSeen: number } {
  apiReads = 1;                             // initial LIST (the watch is one long request)
  const cache = new Map<string, Pod>();     // the "indexer"
  const q = new WorkQueue();
  // the burst arrives faster than workers drain the queue, so keys coalesce.
  // The last 3 watch events are still in flight when the worker runs: the cache lags.
  const delivered = events.slice(0, events.length - 3);
  for (const ev of delivered) { cache.set(ev.name, ev); q.add(ev.name); }
  let reconciles = 0, staleSeen = 0;
  for (let k = q.get(); k; k = q.get()) {
    const latest = cache.get(k)!;           // read from cache, not the server
    if (latest.rv !== server.get(k)!.rv) staleSeen++;
    reconciles++;                           // level-triggered: act on current state of k
  }
  return { reads: apiReads, reconciles, staleSeen };
}

console.log("events in burst:", events.length, "on", new Set(events.map((e) => e.name)).size, "distinct pods");
console.log("naive polling API reads:", naive().toLocaleString("en-US"));
const r = informer();
console.log("informer API reads:", r.reads, "(1 LIST, then 1 open WATCH)");
console.log("reconciles after queue de-dup:", r.reconciles);
console.log("reconciles that saw a stale cache entry:", r.staleSeen, "(writes must carry resourceVersion; a 409 means requeue)");

Output:

events in burst: 1000 on 20 distinct pods
naive polling API reads: 200,000
informer API reads: 1 (1 LIST, then 1 open WATCH)
reconciles after queue de-dup: 20
reconciles that saw a stale cache entry: 3 (writes must carry resourceVersion; a 409 means requeue)

Press Run. Snippets must be self-contained — no network, files, or native modules.

The stale-cache line is the important caveat: informer caches lag the server slightly, so a reconcile can act on old data. That is safe only because writes carry resourceVersions (or use SSA): a stale write fails with a conflict and the key is requeued. Shared informer factories let many controllers in one process share a single watch per resource type.

CRDs and operators: extending the API

A good CRD has a structural schema with validation (OpenAPI plus CEL rules), a status subresource so controllers write status without touching spec, and versioning with conversion if the schema changes. A good operator is level-triggered, idempotent, uses finalizers for external cleanup, writes conditions to status, and never assumes it saw every event.

Decision chart: extending and talking to the control plane.

What happens if you choose an alternative

Pitfalls

• Webhooks that match everything, including kube-system and their own namespace. When the webhook pod dies, it cannot be recreated because its own creation is blocked. Exclude system namespaces.
• Watch history assumptions. Code that resumes a watch from an hour-old resourceVersion must handle 410 Gone by relisting.
• Comparing resourceVersions across resource types. They are only orderable within one resource type, as decimal integers; treat them as opaque otherwise.
• etcd backups that were never restored. A snapshot you have not restored is a hope, not a backup; see Backups That Actually Restore.
• RBAC sprawl. Wildcard verbs on * resources for a CI service account turn a leaked token into cluster admin; see RBAC and role explosion.

Interview Q&A

Check yourself

Elsewhere in the library

These pages stay as they are. This lesson only points at them: Raft Consensus — Leader Election, Log Replication & Safety, Quorums & Majority — Why 2f+1, Read Quorums & Stale Reads, ZooKeeper / etcd Lock Recipes — Ephemeral Nodes, Sessions & Watches, Compare-And-Swap & Optimistic Coordination — When You Do Not Need a Lock, Load Shedding & Admission Control — Drop Early, Protect the Core, RBAC — Roles, Permissions & Role Explosion, Authorization — RBAC, ABAC, ReBAC & Policy Engines, Control Plane & Discovery — xDS-style Config, Endpoints & Convergence, MVCC, Snapshot Isolation & Write Skew.

Go Deeper

• Kubernetes API Concepts: watches, resource versions, streaming lists
• Admission Control in Kubernetes
• Server-Side Apply
• API Priority and Fairness
• Operating etcd clusters for Kubernetes
• etcd system limits
• Custom Resources
• Operator pattern