Thread-per-Connection vs Event Loop vs Async Tasks - Blocking the Loop, C10K & Backpressure

The three models under pressure

flowchart TB
  a["1. Requests arrive faster than downstream drains"]
  b["2. What absorbs the gap?"]
  c["3a. Memory grows, GC pauses, latency climbs"]
  d["3b. Producer pauses, socket stops reading"]
  e["3c. Reject early with 429 or 503"]
  c2["4a. OOM kill or timeouts for everyone"]
  d2["4b. TCP window fills, client slows down"]
  e2["4c. Admitted requests keep good latency"]
  a -->|continues| b
  b -->|unbounded queue or buffer| c
  c -->|continues| c2
  b -->|bounded queue plus backpressure| d
  d -->|continues| d2
  b -->|admission control at edge| e
  e -->|continues| e2

Blocking the event loop (runnable)

A "health check" timer should fire after 10 ms. One 300 ms synchronous loop delays it by about 300 ms, and on a real server that's 300 ms of extra latency for every connection on that loop. Slicing the work lets the timer run on time.

// One CPU-heavy callback delays EVERY other client on an event loop.
// Runs with plain node (setTimeout/Date are global). Times are rounded.
function busy(ms: number) { const end = Date.now() + ms; while (Date.now() < end) {} }

function measure(label: string, work: (done: () => void) => void): Promise<void> {
  return new Promise((resolve) => {
    const scheduled = Date.now();
    // A "health check" timer that should fire after 10ms.
    setTimeout(() => {
      const lag = Date.now() - scheduled - 10;
      console.log(`${label.padEnd(26)} timer lag ~${Math.round(lag / 50) * 50}ms (expected ~0)`);
    }, 10);
    work(() => setTimeout(resolve, 30));
  });
}

(async () => {
  // BAD: 300ms of synchronous CPU in one tick: the loop can't run timers or I/O.
  await measure("blocking 300ms in one tick", (done) => { busy(300); done(); });
  // BETTER: slice the work and yield between slices so timers/I/O interleave.
  await measure("300ms sliced into 5ms steps", (done) => {
    let left = 60;
    const step = () => { busy(5); if (--left > 0) setTimeout(step, 0); else done(); };
    step();
  });
  console.log("best: move CPU work to worker_threads / a separate service");
})();

Output:

blocking 300ms in one tick timer lag ~300ms (expected ~0)
300ms sliced into 5ms steps timer lag ~0ms (expected ~0)
best: move CPU work to worker_threads / a separate service

Common hidden loop-blockers: JSON.parse or JSON.stringify of multi-MB payloads, synchronous fs.*Sync calls, crypto.pbkdf2Sync, big regexes (ReDoS), large array sorts, template rendering, sync logging to a slow disk, and in Python requests or time.sleep inside async def.

Backpressure: bounded vs unbounded (runnable)

# Backpressure: a fast producer and a slow consumer. Without a bound, the queue
# (memory) grows without limit; with a bound, the producer is paused instead.
# asyncio, Python 3.8+.
import asyncio

async def run(maxsize: int) -> int:
    q: asyncio.Queue = asyncio.Queue(maxsize=maxsize)   # 0 = unbounded
    peak = 0
    async def producer():
        nonlocal peak
        for i in range(2000):
            await q.put(i)              # bounded queue: waits here when full
            peak = max(peak, q.qsize())
        await q.put(None)
    async def consumer():
        while (item := await q.get()) is not None:
            if item % 50 == 0:
                await asyncio.sleep(0.001)   # slow downstream (DB, client socket)
    await asyncio.gather(producer(), consumer())
    return peak

unbounded = asyncio.run(run(0))
bounded = asyncio.run(run(64))
print(f"unbounded queue peak depth: {unbounded}  (memory grows with the gap)")
print(f"bounded(64) queue peak depth: {bounded}  (producer paused; same as socket write() -> drain)")

Output:

unbounded queue peak depth: 2000  (memory grows with the gap)
bounded(64) queue peak depth: 64  (producer paused; same as socket write() -> drain)

Press Run. Snippets must be self-contained — no network, files, or native modules.

The same principle shows up at every layer: Node stream.write() returns false when the internal buffer passes highWaterMark, and you wait for 'drain' (or use pipeline()); asyncio StreamWriter.drain(); Netty Channel.isWritable() and write-buffer water marks; gRPC and HTTP/2 flow-control windows; Kafka consumer max.poll.records and pause/resume; and ultimately the TCP receive window, which pushes back on the sender when you stop reading.

C10K to C10M: what changed

• C10K (Dan Kegel, 1999): how do you serve 10,000 concurrent clients on one box? Answer: non-blocking I/O with scalable readiness (epoll, kqueue) instead of a thread or process per client.
• Today: tens or hundreds of thousands of connections per machine are routine for gateways, WebSocket servers and push services. The bottlenecks move to memory per connection (TLS state, buffers), the kernel network stack, and the cost of syscalls.
• C10M thinking: when the kernel itself is the bottleneck, people batch syscalls (io_uring), offload (kTLS, NIC features), or bypass the kernel (DPDK, AF_XDP). Few application teams need this; most need correct backpressure first.

What happens if you choose differently

• Thread pool in front of a slow dependency without timeouts: every thread ends up waiting on the slow call, healthy endpoints are starved too. Use per-dependency bulkheads and timeouts.
• Event loop with one blocking SDK call: everything looks fine at low load; at peak, loop lag spikes, health checks fail, the orchestrator restarts pods, and the remaining pods get more load (cascading failure).
• Async everywhere without limits: asyncio.gather over 100k URLs or a goroutine per message with no semaphore opens 100k sockets, hits fd limits or downstream rate limits. Bound concurrency explicitly.
• Unbounded in-memory queues "to absorb spikes": they turn a latency problem into a memory problem and hide overload until OOM. Bounded queue + reject or block is honest.

How to measure it in production

• Event-loop lag: Node perf_hooks.monitorEventLoopDelay() (p99 above a few tens of ms is a red flag), asyncio debug mode slow_callback_duration, Netty pending tasks per loop.
• Thread pool saturation: active vs max threads, queue depth, rejected tasks.
• Run-queue and context switches: vmstat (r, cs), pidstat -w, Go runtime/metrics scheduler latencies.
• Socket buffers: ss -tmi for send-queue backlog per connection; growing Send-Q means slow clients.
• fd usage: /proc/PID/fd count vs ulimit -n.

Pros and cons

Interview Q&A

Check yourself

Elsewhere in the library

These pages stay as they are. This lesson only points at them: bulkheads, load shedding, performance engineering, the Python GIL, the JavaScript event loop.

Go Deeper

• Node.js: Backpressuring in streams
• Node.js: Don't block the event loop (or the worker pool)
• Node.js worker_threads docs
• Dan Kegel: The C10K problem
• JEP 444: Virtual Threads
• Python asyncio event loop docs