Notes on deterministic replay, verifying fixes, and the bugs that taught us.
The fix passed the crash. It failed 8 of 47 variants.
Kavach records what a service did before it failed, replays it offline, and only accepts a fix that also survives the variants of the incident that break the old build. Here is what that caught, including a bug in a benchmark's own reference solution, and what it can't do yet.
- I graded 26 fixes to six SWE-bench tasks two ways. The graders disagreed on four.Engineering4 min
- The answer key is wrong, the agent got it right, and the benchmark scores them the sameEngineering4 min
- The fix passed the crash. It failed 8 of 47 variants.Product5 min
- What Invariants Should My Service Have?Guides4 min
- How to Read a Kavach Strip ChartGuides4 min
- The Hidden Tests Pass. The Contract Doesn't.Product4 min
- Why Coding Agents Write Narrow Fixes (And Why Logs Won't Save Them)Engineering4 min
- 81.2 Nanoseconds: Designing an Always-On Production Flight RecorderEngineering4 min
- Event SourcingConcepts4 min