Messaging
Part 3 of 6 · WebSockets & MQTTMQTT Essentials — Topics, QoS 0/1/2, Retained Messages & Sessions
MQTT is broker-centric pub/sub: hierarchical topics, hop-scoped QoS 0/1/2, retained last-value, and clean vs persistent sessions. Effective QoS is the min of publish and subscribe. QoS 2 is not Kafka exactly-once.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
MQTT hop vs a durable log
Prefer
Broker pub/sub for live fan-out; log when you need replay
MQTT gives topics, hop-scoped QoS, retained state, and sessions. Use it for devices and presence. Persist high-value events in Kafka (or a time-series DB) when history is the product.
- Publishers never address subscribers — the broker matches filters.
- QoS 1 means duplicates; handlers must be idempotent.
- Retained is last value, not a log.
Alternative
QoS 2 as end-to-end exactly-once, or MQTT instead of Kafka
QoS 2 is exactly once between that client and that broker. A second hop, a bridge, or an HTTP side effect is still at-least-once. Kafka consumer groups and offsets are a different machine.
- Shared subscriptions are competing consumers — not partition assignment.
- Unbounded persistent sessions fill the broker disk.
- Wildcards on publish are invalid; wildcards on subscribe are an ACL hazard.
One retained publish, two delivery stories
Late subscribers get retained. Offline persistent sessions get queued QoS greater than 0. Those are not the same store.
- 1
CONNECT
clientId plus clean start / session expiry. Depth: sessions. - 2
SUBSCRIBE
Topic filter with + or #. Subscription QoS caps what you receive. - 3
PUBLISH QoS 1 retained
Broker PUBACKs. Stores last payload for that topic. Forwards at min of pub and sub QoS. - 4
Subscriber offline
If the session persists, queue QoS greater than 0 until expiry. Retained still overwrites last value. - 5
Do not call it Kafka EOS
This hop is done. Replay, consumer groups, and transactional produce live in the Kafka cluster.
Overview
MQTT is a broker-centric pub/sub protocol. Topics are UTF-8 paths such as sensors/floor1/temp. Publishers never address subscribers; the broker matches filters. Quality of Service, retained messages, and sessions are native — on WebSocket you would build them yourself.
This lesson names the protocol. ACL, Last Will, and bridges are the brokers page. Kafka acks, ISR, and EOS stay in the Kafka messaging cluster — analogy only.
Topics and wildcards
Subscribe wildcards: + is one level; # is multi-level and must be last. Publish never uses wildcards.
Design topic taxonomies for ACL. Flat names force coarse permissions. A later lesson binds tenant ids to prefixes — do not grant # to untrusted clients.
| Filter | Matches | Misses |
|---|---|---|
sensors/+/temp | sensors/f1/temp | sensors/f1/room/temp |
sensors/# | everything under sensors | a sibling alarms/f1 |
sensors/f1/temp | exact | any other path |
QoS 0 / 1 / 2
| QoS | Name | Handshake | Cost |
|---|---|---|---|
| 0 | At most once | Fire-and-forget | Lowest overhead; loss is OK |
| 1 | At least once | PUBACK | Duplicates possible → idempotent handlers |
| 2 | Exactly once on this hop | PUBREC / PUBREL / PUBCOMP | Highest latency and CPU |
Effective delivery is the min of publisher QoS and subscription QoS. Downgrade is a common interview trick: publish 2, subscribe 1 → the subscriber sees QoS 1 (duplicates possible).
Use QoS 0 for high-rate telemetry samples. Use QoS 1 for commands you can make idempotent. Reach for QoS 2 only when that hop truly cannot tolerate duplicates and you accept the handshake.
Cross-link (Kafka — do not re-teach)
Kafka acks (0 / 1 / all), the idempotent producer, and transactional EOS operate on a durable partitioned log with consumer offsets. MQTT QoS is hop-scoped broker ↔ client. Bridge patterns often publish MQTT → Kafka for replay; choose semantics per hop. See Kafka delivery for log replication — here only name the analogy.
MQTT 5 shared subscriptions let consumers compete on a topic (a consumer-group lite). They still are not Kafka partitions: no key-based order, different rebalance.
Retained messages
The broker stores the last retained payload per topic and delivers it immediately on subscribe. Ideal for "current setpoint" / device state.
Retained is not a full history — for history use a log (Kafka topics) or a time-series DB. Clearing: publish a retained empty payload.
Sessions
Clean start (MQTT 5) / clean session (v3.1.1): if false, the broker queues QoS greater than 0 for offline clients (within limits). Persistent sessions enable mobile resume. They explode storage if unbounded — set expiry (MQTT 5 session expiry interval).
| Store | Holds | For whom |
|---|---|---|
| Retained | Last value per topic | Any future subscriber |
| Session queue | Offline QoS greater than 0 | That clientId |
MQTT vs alternatives
- vs WebSocket app pub/sub: MQTT has native topics, QoS, and LWT; on WS you build them.
- vs Kafka: MQTT for live fan-out and devices; Kafka for durable ordered processing.
- vs SSE: MQTT is bidirectional and topic-ACL native; SSE is simpler one-way HTTP.
Sequence
- 1
1 Publisher → 2 Broker
1 CONNECT clientId persist
- 2
3 Subscriber → 2 Broker
"2 CONNECT plus SUBSCRIBE sensors/+/temp QoS1"
- 3
1 Publisher → 2 Broker
"3 PUBLISH sensors/f1/temp QoS1 retained"
- 4
2 Broker → 1 Publisher
4 PUBACK
- 5
2 Broker → 3 Subscriber
5 PUBLISH or retained on late SUB
- 6
3 Subscriber → 2 Broker
6 PUBACK
- 7
3 Subscriber
Step7 offline: queue QoS1 until session expiry
Lesson map
MQTT Essentials — Topics, QoS 0/1/2, Retained Messages & Sessions
MQTT is broker-centric pub/sub: hierarchical topics, hop-scoped QoS 0/1/2, retained last-value, and clean vs persistent sessions. Effective QoS is the min of publish and subscribe. QoS 2 is not Kafka exactly-once.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB p["1 Publisher"] b["2 Broker"] s["3 Subscriber"] p -->|1 CONNECT| b s -->|2 CONNECT plus| b p -->|3 PUBLISH| b b -->|4 PUBACK| p b -->|5 PUBLISH or| s s -->|6 PUBACK| b
Sandbox: match, retain, effective QoS (Python)
Press Run. Snippets must be self-contained — no network, files, or native modules.
Same idea (TypeScript)
Press Run. Snippets must be self-contained — no network, files, or native modules.
Pitfalls
A thermostat publishes QoS 2 retained to sensors/f1/temp. A dashboard subscribes sensors/+/temp at QoS 0. A mobile app uses a persistent session at QoS 1 and goes through a tunnel for 10 minutes. What does each receiver get, and where do duplicates appear? Then clear the retained setpoint without deleting the topic tree.
Interview Q&A
+ vs #?
Answer
- is a single level. # is a multi-level suffix and must be the last token. Neither is legal on PUBLISH.
What is effective QoS?
Answer
The min of the publisher's QoS and the subscription's QoS. A QoS 2 publish to a QoS 0 subscriber is at-most-once at that subscriber.
When QoS 0?
Answer
High-rate telemetry where loss is OK — metrics samples, presence blips you will send again. Not door-lock commands.
Retained vs session queue?
Answer
Retained is the last value for new subscribers on that topic. A session queue is offline durable for that clientId at QoS greater than 0, until session expiry.
Clean start false — what is the risk?
Answer
Broker storage growth. MQTT 5 session expiry interval is the control. Without it, a fleet of abandoned clientIds is a disk incident.
MQTT vs Kafka exactly-once?
Answer
Different layers. MQTT QoS 2 is hop-scoped. Kafka EOS is a transactional produce/consume pipeline on a partitioned log. Cross-link Kafka delivery; do not claim MQTT replaces EOS pipelines.
Can you PUBLISH with #?
Answer
No. Wildcards are subscribe-only. A broker should reject it; an ACL should never need to "allow publish #."
Shared subscriptions in MQTT 5?
Answer
Competing consumers on a topic — a consumer-group lite. Still not Kafka partitions: no sticky key order, different rebalance, no offset log.
How do you clear retained?
Answer
Publish a retained message with an empty payload to that exact topic. The broker deletes the retained copy. Subscribers already online do not rewind history — there is no history.
Why design topics for ACL now?
Answer
building/temp forces everyone to subscribe broadly. tenant/TENANT_ID/device/DEVICE_ID/temp lets the broker bind the token's tenant id to a prefix. Depth: brokers.
Does QoS 1 require idempotent handlers?
Answer
Yes. PUBACK can be lost; the publisher retries; the subscriber may see the command twice. Door locks and payments need an event id.
MQTT vs building pub/sub on WebSocket?
Answer
You can reinvent topics, QoS, and last-will on a WebSocket. MQTT ships them. Use WS when the client is a rich browser UI; use MQTT when the product is topic routing. Depth: choice matrix.