Property-Based & Fuzz Testing — Generators, Shrinking & Invariants
Generators + invariants + shrinking find edge cases example tests miss. Hypothesis (Python) and fast-check (TS); fuzz parsers/protocols.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
The bug is an input you did not think to type
Prefer
Invariant, then shrink
A generator draws hundreds of inputs. The assertion is a relationship that must always hold. A failure shrinks to a minimal counterexample.
- You specify the rule, not every row of a fixture table.
- Shrinking turns a random mess into something you can paste into a unit test.
- Example tests stay, so the happy path is still readable.
Alternative
Only the cases you remembered
Hand-picked examples document intent and skip the empty list, the boundary, and the duplicate key.
- They read like a spec, which is why you keep a few.
- They cannot see past the author's imagination.
- A green file of ten fixtures is not a search of the input space.
Generate, assert, shrink
The source diagram loops back for more trials. This column is one failure path plus the stop condition, which is the interview story.
- 1
Define a generator
Integers in range, lists of records, bytes. Bias toward edges if the library lets you. - 2
Assert the invariant
Round-trip, bounds, or the same answer as a slow oracle. - 3
Shrink on failure
Search for a smaller input that still fails. Report that, plus the seed.
Overview
Example-based tests check the cases you thought of. Property-based tests generate hundreds of inputs and check invariants that must always hold. When one fails, shrinking searches for a smaller input that still fails. Fuzz testing hammers parsers and protocols with malformed input, often with a crash or a sanitizer as the oracle.
Hypothesis is the usual Python library. fast-check is the usual TypeScript library. The sandboxes below do not import either. They run a tiny seeded generator and a shrinker so the loop is visible. The library calls are comments you can paste into a real suite.
| Example-based | Property-based | |
|---|---|---|
| Inputs | Hand-picked | Generated |
| Oracle | Exact expected value | Invariant or relationship |
| Failure output | One case | Minimal shrunk case |
| Strength | Readable specs | Edges humans skip |
| Weakness | Blind spots | Harder oracles, weak for UI |
Properties worth memorizing
- Round-trip.
decode(encode(x))equalsxfor everyxthe generator can build. - Idempotence. Applying the function twice matches applying it once.
- Commutativity. Order of two inputs does not change the result, when the domain says so.
- Metamorphic relation. You do not know the exact output, but you know how outputs of related inputs must connect. Sorting then reversing is the reverse of sorting a reversed list, for a total order.
- Oracle. A slow, obvious implementation matches the fast one on generated inputs. The cart total below is this idea: the property is "non-negative for non-negative lines," which a buggy implementation can still violate once shrinking finds the shape.
A sort earns three properties even without a hand-built expected list: same length, sorted order, and a permutation of the input.
Flow
- 1
1. Define a generator
- next2. Draw a sample
- 2
2. Draw a sample
- next3. Assert the invariant
- 3
3. Assert the invariant
- next4. On fail, shrink input
- 4
4. On fail, shrink input
- next5. Minimal counterexample
- 5
5. Minimal counterexample
- next6. On pass, stop at budget
- 6
6. On pass, stop at budget
Lesson map
Property-Based & Fuzz Testing — Generators, Shrinking & Invariants
Generators + invariants + shrinking find edge cases example tests miss. Hypothesis (Python) and fast-check (TS); fuzz parsers/protocols.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB a["1. Define a generator"] b["2. Draw a sample"] c["3. Assert the invariant"] d["4. On fail, shrink input"] a -->|1. Define a generator| b b -->|2. Draw a sample to 3. Assert the invariant| c c -->|3. Assert the invariant to| d
When not to use them
Skip property tests for UI layout. You do not have a stable invariant for "looks aligned," and screenshots flake. Skip them for nondeterministic systems unless you control the seed, the clock, and the ordering. A property that calls datetime.now or compares sets by iteration order will fail at random. Freeze the clock, seed the generator, and sort before you assert. That fix is the flake page.
Also skip tautologies. assert result == result generates a lot of inputs and tests nothing. If you cannot state a relationship that a wrong function would break, write an example instead.
Fuzzing is the cousin for parsers, codecs, JWT and URL decoders, and any untrusted bytes. The oracle is often "does not crash" or "sanitizer stays quiet," not a rich invariant. Property-based tests shine when you can state the invariant. Fuzzing shines when the input is hostile and the bug is memory safety or a hang. They complement each other. Neither replaces the one example that documents the feature.
A shrink you can run
The function under test drops the last number whenever the list is longer than three. Short examples pass. A seeded search finds a longer list, then shrinks it. Hypothesis would do this with @given and would print a seed so you can replay a CI-only failure with reproduce_failure. fast-check would do it with fc.assert and fc.property. Those imports are comments, because this sandbox cannot load them.
Press Run. Snippets must be self-contained — no network, files, or native modules.
Press Run. Snippets must be self-contained — no network, files, or native modules.
A cart total over non-negative quantity and unit price is the doc's other invariant: the sum stays non-negative. Hypothesis would build Line values with st.builds and lists with st.lists. fast-check would use fc.array and fc.record with integer ranges. Keep a single example, ten percent off a known cart, next to that property so a reader can see the feature without reading the generator.
Interview Q&A
What is shrinking?
Answer
After a random input fails, the library searches for a smaller input that still fails. Smaller means a shorter list, a number closer to zero, or fewer interesting branches. You debug the minimal counterexample, not the first garbage value the generator drew. Then you pin that value as an example test so the bug cannot return quietly.
What three properties would you write for a sort?
Answer
Same length as the input. The output is ordered. The output is a permutation of the input, which you can check with a frequency map. Those three break the usual bugs: dropping an element, failing to order, and inventing a value. You do not need a hand-written expected array for each input.
How is property-based testing different from fuzzing?
Answer
Property-based tests pair a generator with an invariant and a shrinker. Fuzzing throws malformed bytes at a parser or protocol and usually oracles on crashes, hangs, or sanitizers. Use properties when you can state the rule. Use fuzzing on untrusted parsers and codecs. A round-trip property on a decoder is often both.
Why would a property test flake?
Answer
It read the wall clock, compared an unordered set by accident, or depended on hash iteration. Freeze time, pass a seed, and sort before asserting. A property suite that fails without a code change is a flake, and retries will hide the invariant you meant to lock. Replay with the printed seed.
Do property tests replace examples?
Answer
No. Keep a few readable examples so the next person sees the intended behavior in one screen. Let properties hunt the edges those examples miss. Deleting the examples because "the property covers it" makes the suite true and unreadable.
What is a metamorphic relation?
Answer
A relationship between the outputs of related inputs, used when you do not know the exact answer. If you add a constant to every point, a translation-invariant function moves by that constant. If you only assert output > 0, a lot of wrong functions still pass. The relation has to be something the bug would break.
Where do you fuzz first?
Answer
Parsers, codecs, JWT and URL decoders, and any API that accepts untrusted bytes. The oracle can be "no crash" while you invent richer invariants. Do not start by fuzzing a React layout. You will not get a shrink that explains a pixel.
A failure only happens in CI. How do you replay it?
Answer
Hypothesis prints a seed and a @reproduce_failure decorator. fast-check prints the seed and the shrunk value. Rerun with that seed locally. If you cannot reproduce, you do not have a property failure yet. You have nondeterminism, which is the flake page.
What does a bad oracle look like?
Answer
An assertion that the buggy function still satisfies. total == total, or len(result) >= 0, never fails. Write the property by asking which wrong implementation would slip through. If the answer is "most of them," the oracle is too weak.
What do you say in the first minute?
Answer
Examples check cases you thought of. Properties generate inputs and check invariants. Shrinking finds a minimal counterexample. Hypothesis and fast-check are the practical libraries. Fuzz parsers. Do not property-test UI layout or an unseeded clock.
Pitfalls
- Replacing every example with a property and losing the readable spec.
- Writing a tautology that cannot fail.
- Letting the generator call the network or the system clock.
- Comparing unordered collections without sorting.
- Using property tests for screenshot layout.
- Retrying a shrunk failure instead of replaying the seed.
For a function that merges two permission sets, state three properties that a bug would break. Then name one assertion you will not write because every implementation would pass it. Say whether a fuzzer belongs here or a property test does.