Engineering practices
Part 4 of 6 · Interview Debugging & Implementation — Reproduce, Trace, Fix, ShipFix, Verify & Roll Back — Smallest Change & Proof
The smallest correct diff, a characterization test, proof that the same query went quiet, and a rollback with a blast radius.
- 1Gist
- 2Maps
- 3Q&A
- 4Sandbox
Voice readout needs Web Speech Synthesis in this browser.
Question ladder
L1
What is the smallest correct diff?
Answer
The minimum change that makes the failing request succeed and leaves the contract intact.
L2
What is a characterization test?
Answer
A test that locks what the system does today, including the wrong status, before you change it.
L3
How do you know you are done?
Answer
The original signal is quiet for the repro, the tests you added pass, and rollback is stated.
L4
Why rerun the same query?
Answer
A new dashboard can be green while the original failing filter is unchanged. Proof is the same signal.
L5
What are the rollback options?
Answer
Flag off, revert or the prior artifact, or a config restore. Each has a different speed and a different forgotten-state risk.
L6
The interviewer wants a bigger refactor. Then what?
Answer
Acknowledge the debt, propose a follow-up, and land the minimal fix unless they explicitly score design over recovery.
L7
The fix works locally and you cannot redeploy. Then what?
Answer
Show the diff and the test. Describe the verify query you would run after deploy. Ask if they want the feature next.
Failure modes
Rewrite instead of a mapping fix
A new retry framework starts while checkout still maps every upstream failure to a generic 500.
Proof on a different graph
A fresh dashboard looks calm. The original PaymentTimeout filter was never rerun.
Rollback with no blast radius
You can revert the commit and still cannot say which tenants already wrote, or whether a cache is stale.
Timeout bump as a disguise
Raising the client timeout makes the 500 stop and leaves the slow dependency unnamed.
Misconceptions
The elegant redesign is what senior looks like.
Senior in this slice is a diff you can prove and undo. The redesign is a follow-up ticket.
A green unit test replaces the original query.
The test locks the behavior you encoded. The query checks the repro the interviewer handed you.
Rollback is only a git revert.
A flag and a config restore are faster when they exist. A revert is clearer when they do not.
Interviewer traps
Redesign the CI pipeline as part of the fix.
Say gated deploy or flag off, and move on. Pipeline design is a different series.
Teach Terraform state while discussing rollback.
Name the artifact or the flag. Infrastructure change is out of this slice.
Design scenario
Same prompt for every reader.
Requirements
A characterization of today's 504 to 500 mapping, the smallest new mapping, the same request or query as proof, and a rollback sentence.
Traffic / scale
One repro request. Not a fleet-wide canary design.
Latency
Do not retune every timeout in the client. If latency is the product bug, say so and keep the diff small.
Consistency
200 still means paid. A 409 must not become a retryable 503 by accident.
Availability
Flag off or the previous artifact returns the old behavior. Say who already took the new path.
Failure assumptions
- You may not get a redeploy in the room.
- A timeout increase would also make the demo green.
- The interviewer may ask for a refactor.
Constraints
- No new HTTP client library.
- No database migration.
Prompt
You have confirmed a payment 504 that checkout maps to a generic 500. Fix the mapping, prove it, and say how you undo it.
API
What status and error string should a 504 produce after the fix?
Data
Which assertion is red before the edit and green after?
Architecture
What is the rollback, and which tenants does it not reach instantly?
What you land in the remaining minutes
Prefer
One mapping, then the same query
Today a 504 becomes a generic 500. After the diff, a 504 becomes a 503 with PaymentTimeout. The original filter is the proof.
- The characterization fails on the old function and passes on the new one.
- 200 and 409 stay distinct.
- Rollback is a sentence, not a project.
Alternative
A retry framework before the mapping
New client, new policy, no proof the original 500 path changed.
- Time runs out with the bug still generic.
- The feature never starts.
- Review surface is the whole client, not the status line.
From a confirmed cause to a quiet signal
A new chart is not the definition of done.
- 1
Lock today's behavior
A characterization test asserts the wrong mapping first, so the bug is reproducible in code. - 2
Smallest diff
Change the mapping or the one flag. Do not extract a framework. - 3
Same signal
Rerun the original filter, the request id, or the test that was red. - 4
Rollback sentence
Flag, revert, or prior artifact, and which tenants still see the new path.
Smallest correct diff
Ask what minimum change makes the failing request succeed without breaking the contract.
| Change | Example | Risk |
|---|---|---|
| Config, timeout, or flag | Raise a client timeout, or disable the bad path | Low code risk, and it may hide the cause |
| One-line logic | A null check, or a status mapping | Low when tested |
| Extract and redesign | A new retry framework during the bug fix | High, and usually the wrong use of this room |
The redesign runs you out of time. The smallest fix is shippable proof and may leave a follow-up for retries or a circuit breaker. Those patterns already have pages. Do not rebuild them here.
Decisions
- 1
1 Confirmed root cause
- next2 Test locks today's bug
- 2
2 Test locks today's bug
- next3 Smallest diff
- 3
3 Smallest diff
- next4 Replay the same query
- 4
4 Replay the same query
- next5 Signal quiet?
- ?
5 Signal quiet?
- no3 Smallest diff
- yes6 State the rollback
- 6
6 State the rollback
- next7 Feature is next
- 7
7 Feature is next
Lesson map
Fix, Verify & Roll Back — Smallest Change & Proof
The smallest correct diff, a characterization test, proof that the same query went quiet, and a rollback with a blast radius.
Architecture. Architecture
Select a node to see why it exists, or an edge to see the protocol, direction, effect, and consequence.
Mermaid export
flowchart TB c["1 Confirmed root cause"] t["2 Test locks today's bug"] f["3 Smallest diff"] v["4 Replay the same query"] c -->|1 Confirmed root cause| t t -->|2 Test locks today's bug| f f -->|3 Smallest diff to 4 Replay the same query| v
Characterization
A characterization test asserts what the system does today, even when that behavior is wrong. Example: given payments returns 504, checkout maps to 500 with PaymentError. After the fix, the same input expects 503 and PaymentTimeout, so a client can tell a timeout from a conflict. Pros: you proved you understood the bug. Cons: it costs minutes. Skip it only when they push for speed and the fix is one obvious line, and say the test you would add.
Proof is the same signal
Do not invent a dashboard. Rerun:
- The filter for PaymentTimeout on the fixed build, or against the local server.
- Or the single request id replay.
- Or the unit test that was red.
Say: done means that query returns no matching errors for the repro case. If they care about burn, mention the budget in one clause and stay on SLOs, error budgets, and tracing.
Rollback and blast radius
| Rollback | Pros | Cons |
|---|---|---|
| Feature flag off | Fast and targeted | Needs a flag, and config can lag |
| Revert or prior artifact | Clear history | Slower, and may revert unrelated commits |
| Config restore | Fast for a timeout or a pool knob | Easy to forget when config is generated |
Say which tenants, which regions, whether writes were dual-written, and whether a cache is stale. That is the blast radius. It is not an infrastructure lecture.
Sandbox
The old function is the characterization. The new function is the smallest mapping fix.
Problem504 becomes a generic 500 today. After the fix, 504 and 408 become 503 PaymentTimeout, 409 stays 409, and 200 stays 200.
ExpectedThe old function returns (500, PaymentError) for 504. The new function returns (503, PaymentTimeout) for 504 and (200, None) for 200.
Edge cases
- 409 must not look retryable.
- A 500 from upstream is a 502, not a timeout.
- Test: today's bug is locked
map_payment_status(504) == (500, 'PaymentError') - Test: 504 is a timeout after the fix
map_payment_status_fixed(504) == (503, 'PaymentTimeout') - Test: 408 matches 504
map_payment_status_fixed(408) == (503, 'PaymentTimeout') - Test: 409 is not retried as a timeout
map_payment_status_fixed(409) == (409, 'PaymentConflict') - Test: success is unchanged
map_payment_status_fixed(200) == (200, None)
Press Run. Snippets must be self-contained — no network, files, or native modules.
ProblemShow the generic 500, then the mapping that distinguishes timeout, conflict, and other upstream failures.
ExpectedmapPaymentStatus(504) is [500, PaymentError]. mapPaymentStatusFixed(504) is [503, PaymentTimeout].
Edge cases
- 200 stays a pair of 200 and null.
- An unknown 500 becomes 502.
- Test: old mapping is generic
JSON.stringify(mapPaymentStatus(504)) === JSON.stringify([500, 'PaymentError']) - Test: new mapping names the timeout
JSON.stringify(mapPaymentStatusFixed(504)) === JSON.stringify([503, 'PaymentTimeout']) - Test: conflict stays 409
JSON.stringify(mapPaymentStatusFixed(409)) === JSON.stringify([409, 'PaymentConflict'])
Press Run. Snippets must be self-contained — no network, files, or native modules.
Interview Q&A
How do you know you are done?
Answer
The original failing signal is quiet for the repro. Tests you added or updated pass. Rollback is a sentence you have already said.
The interviewer wants a bigger refactor.
Answer
Acknowledge the debt and name a follow-up. Land the minimal fix unless they explicitly score design over recovery. The hub is where you ask which slice they want scored.
The fix works locally and you cannot redeploy.
Answer
Show the diff and the test. Describe the verify query you would run after deploy. Ask whether they want the feature next.
What do you not do in this slice?
Answer
Rewrite CI, redesign infrastructure state, or migrate the database. Say "gated deploy" or "flag off" and move on.
Why keep the old function in the example?
Answer
So the characterization stays visible. In a real diff you replace the body and keep the test that used to expect the bug, updated to the new contract.
Is raising the timeout a fix?
Answer
It can be the smallest change, and it can also hide the slow dependency. Say that risk out loud if you choose it. Prefer the mapping when the bug is a wrong status.
How does an error budget show up here?
Answer
One clause: these 500s spend budget. The policy and the burn windows are SLOs, error budgets, and tracing. They do not change the diff.
What do you hand the feature slice?
Answer
A quiet repro, the contract you just preserved (200, 409, timeout), and the rollback you can still pull if the feature misbehaves.
Pitfalls
- A new client library as the bugfix.
- Proof on a dashboard you invented during the interview.
- A revert story that ignores stale caches and in-flight writes.
- Collapsing 409 and 504 into the same retryable status.
- Skipping the sentence about who is still exposed.
In one breath: the old 504 mapping, the new 504 mapping, the query you rerun, and the rollback. Then name one follow-up you will not start in this room.
Go deeper
- Characterization tests are the move from Working Effectively with Legacy Code: lock behavior, then change it.
- CodeDeploy rollback is one concrete undo. A flag is the other.
- Budget policy, if they ask, is the SRE workbook chapter and the SLO series, not a second fix.
Next: Implement the feature.