Protobuf Schemas — Fields, Wire Types & Compatibility
Protobuf's wire identity is the field number, not the field name. Types, optional / repeated / map, proto3 defaults, reserved, and wire types determine whether two binaries can talk forever. Interviewers probe: can you add a field safely? Why never reuse numbers? What goes wrong with JSON mapping?
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
What is the identity of a field?
Prefer
Field number + wire type (Protobuf)
The tag is (field_number << 3) | wire_type. Old binaries ignore unknown numbers. Rename is wire-safe; reuse is silent corruption. buf breaking detection encodes the discipline.
- Add a new number; deploy readers first; then writers.
- Delete by reserving the number and name so protoc blocks accidents.
- Avro keys off names + writer schema; JSON Schema keys off property names — different evolution story.
Alternative
Property names or “just add a JSON key”
Looks easy. JSON mapping then surprises you: lowerCamelCase, int64 strings, omitted defaults that are not null. Reusing a number because “the name changed” corrupts old clients.
- Changing type in place usually changes the wire encoding.
- Enum 0 must stay unknown/unspecified; new values need unknown handling.
- Public REST versioning is a different lever — see the API Design cluster.
Safe evolution of User
Readers that ignore unknowns first, then writers that set the new field. Never reuse the old number.
- 1
Need new data
Product wants a field the old User message does not have. - 2
Allocate a number
Unused field number. Prefer 1–15 for hot fields (1-byte tags). - 3
Add optional / message
Do not renumber. Do not change type in place. - 4
Deploy readers
New code ignores unknown fields; old data still decodes. - 5
Deploy writers
Only then set the new field. If deleting: reserved the old number forever.
Overview
Protobuf's wire identity is the field number, not the field name. Types, optional / repeated / map, proto3 defaults, reserved, and wire types determine whether two binaries can talk forever.
Interviewers probe: can you add a field safely? Why never reuse numbers? What goes wrong with JSON mapping?
Field numbers are forever
| Rule | Why |
|---|---|
| Numbers 1–15 use 1-byte tags | Put hot fields here |
| Never reuse a number | Old binaries mis-decode new meaning |
reserved deleted numbers/names | Compiler blocks accidents |
| Names can change | Wire ignores names; JSON/docs care |
Types & presence (proto3)
| Kind | Behavior | Pitfall |
|---|---|---|
| Scalar (int32, string…) | Default zero / empty if unset | Cannot distinguish unset vs default without optional |
optional scalar | Presence tracked | Use when zero is meaningful |
repeated | List; packed for numerics by default | Order preserved; duplicates allowed |
| map of K to V | Unordered logical map | Keys limited types; not truly ordered |
oneof | At most one member set | Changing members needs care |
| Enum | Integer on wire | Always keep 0 unknown / unspecified |
required in proto2 was a deployment foot-gun; proto3 dropped it. Enforce in application validation.
Wire types (interview cheat sheet)
| Wire type | Used for |
|---|---|
| 0 Varint | int32/64, uint, bool, enum |
| 1 64-bit | fixed64, sfixed64, double |
| 2 Length-delimited | string, bytes, embedded messages, packed repeated |
| 5 32-bit | fixed32, float |
Unknown fields: parsers should preserve them for proxying / round-trip (implementation-dependent; modern runtimes do).
Compatibility matrix
| Change | Safe? | Notes |
|---|---|---|
| Add new field with new number | Yes | Old ignore; new optional |
| Rename field | Wire-safe | Breaks JSON / docs |
| Change field type in place | Usually no | Different wire encoding |
| Delete field, reserved number | Yes | Prevent reuse |
| Reuse number for new meaning | Never | Silent corruption |
| Expand enum with new value | Careful | Old must handle unknown |
| Tighten validation in app | OK | Not a schema break |
Forward compatible: new code reads old data.
Backward compatible: old code reads new data (ignores unknowns).
Flow
- 1
1 Need new data on User
- next2 Allocate unused field no
- 2
2 Allocate unused field no
- next3 Add optional or message
- 3
3 Add optional or message
- next4 Deploy readers first
- 4
4 Deploy readers first
- next5 Deploy writers next
- 5
5 Deploy writers next
- next6 If deleting: reserved
- 6
6 If deleting: reserved
- next7 Never reuse that number
- 7
7 Never reuse that number
Lesson map
Protobuf Schemas — Fields, Wire Types & Compatibility
Protobuf's wire identity is the field number, not the field name. Types, optional / repeated / map, proto3 defaults, reserved, and wire types determine whether two binaries can talk forever. Interviewers probe: can you add a field safely? Why never reuse numbers? What goes wrong with JSON mapping?
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB a["1 Need new data on User"] b["2 Allocate unused field no"] c["3 Add optional or message"] d["4 Deploy readers first"] a -->|1 Need new data on User| b b -->|2 Allocate unused field no| c c -->|3 Add optional or message| d
JSON mapping pitfalls
- Field names become lowerCamelCase by default — surprises for snake_case APIs.
- int64 / uint64 often strings in JSON to avoid JS precision loss.
- Bytes are base64.
- Default values may be omitted — do not treat omission as null semantics unless
optional.
Comparative: Protobuf vs Avro vs JSON Schema
| Protobuf | Avro | JSON Schema | |
|---|---|---|---|
| Identity | Field number | Name + writer schema | Property names |
| Evolution | Number discipline | Reader/writer schema pair | Often looser |
| Payload | Compact binary | Compact + optional schema | Text |
| RPC home | gRPC | Kafka ecosystem | REST / OpenAPI |
Kafka registry / DLQ depth: schema evolution & DLQ — this page is the Protobuf wire, not poison-message routing.
Sandbox: tag + length-delimited string (Python)
Toy encoder for string name = 1. Real systems use google.protobuf or betterproto — this teaches the wire idea. Length must stay under 128 so the varint is one byte.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Same toy wire (TypeScript)
Press Run. Snippets must be self-contained — no network, files, or native modules.
Pitfalls
User has string name = 1. You need email. Which number? Who deploys first? What do you reserved if you later delete email?
Interview Q&A
Why are field numbers immutable?
Answer
They are the on-wire keys. Reuse makes old clients interpret new bytes as the old type.
What does reserved do?
Answer
Tells protoc to reject reuse of deleted field numbers or names.
proto2 required vs proto3?
Answer
required was a deployment foot-gun; proto3 dropped it. Enforce in application validation.
Are maps ordered?
Answer
No — treat as unordered. If order matters, use repeated pairs.
How do you delete a field safely?
Answer
Stop writing it, reserve the number/name, keep readers tolerant of absence.
Why might JSON show int64 as a string?
Answer
JavaScript Number is IEEE-754 double; large integers lose precision.
Packed repeated — what is it?
Answer
Numeric repeated fields encoded as one length-delimited blob of varints — denser on the wire.
Can you change int32 to int64?
Answer
Often wire-compatible for varints in range, but semantic/API risk — prefer a new field.