The fix passed the crash. It failed 8 of 47 variants.

Here's a bug I've seen in production more times than I'd like. A wallet service folds deposit, withdraw and transfer events into balances. One day upstream sends a deposit with a null amount, and the handler dereferences a nil pointer.

Hand that crash to a coding agent, and what comes back? Usually something like this:

if ev.Type == "deposit" && ev.Amount == nil {
    return reject(env, ev, "missing amount")
}

It handles the input that crashed. The test written from the incident goes green, and a reviewer sees a null check and approves. Then the next transfer with a null amount crashes the same way, because the fix only guards deposits.

Is the model being dumb? I don't think so. It got one failing input, and the shortest way to make it pass was a check for that one input. A log file can't push back on that, and an agent can't execute a log. So I built something it can execute.

What Kavach does#

Kavach¹ is a flight recorder and a replayer for services that process one input at a time. Your handler reads the clock, randomness, config and gateways through an SDK, which hands every event to a separate recorder process. When a step panics, errors or breaks an invariant you declared, the recorder writes a fixture: every input and read, every output, and how it failed.

What do you do with a fixture? Three things:

It's Apache 2.0, pre-alpha, with SDKs for 12 languages.

The ledger, run for real#

Here's the README's demo on my laptop, output as printed. The eighth event is the null deposit:

$ ./ledger-old -in events.jsonl
panic: runtime error: invalid memory address or nil pointer dereference
 
$ kavach inspect kavach/fixtures/ledger-*.kavach
    26  input     file:events.jsonl @ 7  {"id":"evt-007","type":"transfer","account":"carol","to":"alice","amount":400}
    27  clock     2026-10-26T13:16:45.484889Z
    28  rand      8 bytes  4de1e08348cb9809
    29  output    ledger.entries  {"txn":"4de1e08348cb9809","event":"evt-007","account":"carol","delta":-400,...
    ...
    32  input     file:events.jsonl @ 8  {"id":"evt-008","type":"deposit","account":"bob","amount":null}
    33  marker    panic  runtime error: invalid memory address or nil pointer dereference

What happens when you replay that fixture on the build that crashed?

$ kavach replay kavach/fixtures/ledger-*.kavach --bin ./ledger-old
result    still_failing@32
wall      10.1 ms (including process start)

It fails again, at seq 32. Replay never reads the system clock or the random generator, so the IDs and timestamps match production's. That's the whole trick.

Now the deposit-only fix:

$ kavach diff kavach/fixtures/ledger-*.kavach --old ./ledger-old --new ./ledger-partial
old       still_failing@32
new       fixed
variants  47 of 56 reproduce the incident on the old build · 39 of those pass on the new build
  ✗  7 failing input (seq 32): "type" "deposit" → "transfer"
       still_failing@23: panic: runtime error: invalid memory address or nil pointer dereference
  ✗ 35 failing input (seq 32): "type" "deposit" → "withdraw"
       still_failing@23: panic: runtime error: invalid memory address or nil pointer dereference
  ...
regressions 46 of 46 match the old build · 48 variants of 4 earlier inputs resembling the failing one
verdict   variant_failed(7)@23
wall      861.3 ms for 199 replays

The recorded incident passes. The variants don't: the same event as a transfer still panics, and so does a withdrawal. Variant 7 is saved as its own fixture for the agent to replay. And the fix that checks the amount on every event type?

$ kavach diff kavach/fixtures/ledger-*.kavach --old ./ledger-old --new ./ledger-new
variants  47 of 56 reproduce the incident on the old build · 47 of those pass on the new build
regressions 46 of 46 match the old build · 48 variants of 4 earlier inputs resembling the failing one
verdict   fixed

All 47 pass, in under a second.

Which variants count?#

Generating variants is easy. Diff changes or removes fields of the failing input, moves it earlier, drops or redelivers earlier inputs, and shifts the clock. Which of them are evidence?

If a mutation makes the old build fail some other way, what does that say about the fix? Nothing. So diff keeps only the variants that break the old build the way production broke: 47 of 56 here, and the new build must pass all of them. With fewer than 10, the verdict is unverified, not fixed.

Rejecting every event also makes the crash go away, right? So diff also mutates earlier inputs that look like the failing one, here the other deposits, and the new build has to match the old build's outputs on them. It matched on all 46.

Each verdict tells the agent what to do next, and the agent guide³ has the loop.

The hidden tests passed. The reference solution was wrong.#

The ledger is a demo with a bug I planted. What does this do on code I didn't write? I screened 613 public benchmark problems from DeepSWE, SWE-bench Multilingual and Multi-SWE-bench, and built 11 tasks on Redis, Valkey, axum, kombu, httpx, go-genai and updo, each graded by the upstream hidden test.

Features get a contract: traffic recorded before the feature, invariants from the spec, and a person's approval before any code. Every feature step must change, every other step must replay byte for byte, and the invariants must hold.

On the five DeepSWE feature tasks, every reference implementation passes its contract except one. These builds pass the hidden tests and fail the contract:

I didn't expect the first row. In DeepSWE's httpx reference, the function that stores a cookie never runs the limit check, so the store grows past the spec's limits. The hidden tests never set enough cookies to notice. I wrote it up in The answer key is wrong.

The other three rows are builds I wrote on purpose, to find what each task's hidden tests miss. They're plausible mistakes, not ones an agent made. The results and method⁴ list every build, including the wrong ones the hidden tests do catch.

Agents, with and without it#

Does any of this help an agent? The benchmark runs OpenCode with the same prompt in every arm; only the model and Kavach's MCP tools change, and the hidden test grades. The first trials, on four Redis incidents with small models:

That's 25 trials per arm: enough to show the harness works, not enough to claim an effect size. Every run is public at benchmark.kavachlabs.dev⁵.

What it misses#

Where does it get things wrong?

How it compares#

Try it#

To see it without installing anything, open try.kavachlabs.dev⁷. It has saved agent runs, with and without Kavach, such as an agent fixing redis-11734⁷ and replaying the crash on each attempt.

To run it on your own code:

curl -fsSL https://www.thekavachlabs.com/install.sh | sh

claude mcp add kavach -- kavach mcp

codex mcp add kavach -- kavach mcp

{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "kavach": { "type": "local", "command": ["kavach", "mcp"] }
  }
}

The install script also adds the four Kavach skills to Claude Code, and to Codex and OpenCode when it finds them. The README¹ walks through the ledger demo, and there's a Kafka version of it that replays with no broker running.

What I'm looking for#

I spent five years on financial systems where every serious incident was a replay problem nobody could actually run. Kavach is my attempt to make that replay something you can run, and something an agent has to pass before a person reviews its fix.

What it needs now is an incident that isn't mine. Do you run a service that consumes from a queue or a log, and have you had a failure you couldn't reproduce? I'd like to record it with you and see whether the variants find what your tests didn't. Write to hello@thekavachlabs.com, or open an issue on GitHub¹.