# The fix passed the crash. It failed 8 of 47 variants.

> Kavach records what a service did before it failed, replays it offline, and only accepts a fix that also survives the variants of the incident that break the old build. Here is what that caught, including a bug in a benchmark's own reference solution, and what it can't do yet.

Koustav Chowdhury · October 27, 2026 · Product

Here's a bug I've seen in production more times than I'd like. A wallet service folds deposit, withdraw and transfer events into balances. One day upstream sends a deposit with a null amount, and the handler dereferences a nil pointer.

Hand that crash to a coding agent, and what comes back? Usually something like this:

```go
if ev.Type == "deposit" && ev.Amount == nil {
    return reject(env, ev, "missing amount")
}
```

It handles the input that crashed. The test written from the incident goes green, and a reviewer sees a null check and approves. Then the next transfer with a null amount crashes the same way, because the fix only guards deposits.

Is the model being dumb? I don't think so. It got one failing input, and the shortest way to make it pass was a check for that one input. A log file can't push back on that, and an agent can't execute a log. So I built something it can execute.

![Kavach on the ledger demo: recording a nil-pointer crash, replaying it with no broker running, and rejecting the deposit-only fix across 56 variants](https://www.thekavachlabs.com/blog/replay-the-crash-prove-the-fix/ledger-demo.gif "The ledger demo, start to finish")

## What Kavach does

[Kavach](https://github.com/kavachlabs/kavach) is a flight recorder and a replayer for services that process one input at a time. Your handler reads the clock, randomness, config and gateways through an SDK, which hands every event to a separate recorder process. When a step panics, errors or breaks an invariant you declared, the recorder writes a fixture: every input and read, every output, and how it failed.

What do you do with a fixture? Three things:

```markdown
**kavach replay** runs your own build against the fixture and serves every read from it. No broker, no database, no network.

**kavach diff** judges a fix. It replays the incident on the old build and the new one, then replays variants of the incident on both.

**kavach mcp** serves both to Claude Code, Codex, OpenCode or any MCP client, so the agent gets the verdict as JSON.

Recording adds a median 81.2 ns per event, measured with the command in [BENCHMARKS.md](https://github.com/kavachlabs/kavach/blob/main/BENCHMARKS.md).
```

It's Apache 2.0, pre-alpha, with SDKs for 12 languages.

## The ledger, run for real

Here's the README's demo on my laptop, output as printed. The eighth event is the null deposit:

```text
$ ./ledger-old -in events.jsonl
panic: runtime error: invalid memory address or nil pointer dereference

$ kavach inspect kavach/fixtures/ledger-*.kavach
    26  input     file:events.jsonl @ 7  {"id":"evt-007","type":"transfer","account":"carol","to":"alice","amount":400}
    27  clock     2026-10-26T13:16:45.484889Z
    28  rand      8 bytes  4de1e08348cb9809
    29  output    ledger.entries  {"txn":"4de1e08348cb9809","event":"evt-007","account":"carol","delta":-400,...
    ...
    32  input     file:events.jsonl @ 8  {"id":"evt-008","type":"deposit","account":"bob","amount":null}
    33  marker    panic  runtime error: invalid memory address or nil pointer dereference
```

What happens when you replay that fixture on the build that crashed?

```text
$ kavach replay kavach/fixtures/ledger-*.kavach --bin ./ledger-old
result    still_failing@32
wall      10.1 ms (including process start)
```

It fails again, at seq 32. Replay never reads the system clock or the random generator, so the IDs and timestamps match production's. That's the whole trick.

Now the deposit-only fix:

```text
$ kavach diff kavach/fixtures/ledger-*.kavach --old ./ledger-old --new ./ledger-partial
old       still_failing@32
new       fixed
variants  47 of 56 reproduce the incident on the old build · 39 of those pass on the new build
  ✗  7 failing input (seq 32): "type" "deposit" → "transfer"
       still_failing@23: panic: runtime error: invalid memory address or nil pointer dereference
  ✗ 35 failing input (seq 32): "type" "deposit" → "withdraw"
       still_failing@23: panic: runtime error: invalid memory address or nil pointer dereference
  ...
regressions 46 of 46 match the old build · 48 variants of 4 earlier inputs resembling the failing one
verdict   variant_failed(7)@23
wall      861.3 ms for 199 replays
```

The recorded incident passes. The variants don't: the same event as a transfer still panics, and so does a withdrawal. Variant 7 is saved as its own fixture for the agent to replay. And the fix that checks the amount on every event type?

```text
$ kavach diff kavach/fixtures/ledger-*.kavach --old ./ledger-old --new ./ledger-new
variants  47 of 56 reproduce the incident on the old build · 47 of those pass on the new build
regressions 46 of 46 match the old build · 48 variants of 4 earlier inputs resembling the failing one
verdict   fixed
```

All 47 pass, in under a second.

## Which variants count?

Generating variants is easy. Diff changes or removes fields of the failing input, moves it earlier, drops or redelivers earlier inputs, and shifts the clock. Which of them are evidence?

If a mutation makes the old build fail some other way, what does that say about the fix? Nothing. So diff keeps only the variants that break the old build the way production broke: 47 of 56 here, and the new build must pass all of them. With fewer than 10, the verdict is unverified, not fixed.

Rejecting every event also makes the crash go away, right? So diff also mutates earlier inputs that look like the failing one, here the other deposits, and the new build has to match the old build's outputs on them. It matched on all 46.

Each verdict tells the agent what to do next, and the [agent guide](https://github.com/kavachlabs/kavach/blob/main/AGENTS.md) has the loop.

## The hidden tests passed. The reference solution was wrong.

The ledger is a demo with a bug I planted. What does this do on code I didn't write? I screened 613 public benchmark problems from DeepSWE, SWE-bench Multilingual and Multi-SWE-bench, and built 11 tasks on Redis, Valkey, axum, kombu, httpx, go-genai and updo, each graded by the upstream hidden test.

Features get a contract: traffic recorded before the feature, invariants from the spec, and a person's approval before any code. Every feature step must change, every other step must replay byte for byte, and the invariants must hold.

On the five DeepSWE feature tasks, every reference implementation passes its contract except one. These builds pass the hidden tests and fail the contract:

```markdown
| Task | Build | Hidden tests | kavach contract check |
| --- | --- | --- | --- |
| httpx cookie store | DeepSWE's reference solution | pass | invariant_violated(cookie-limits)@190 |
| httpx cookie store | one store shared by every client | pass | regression@13 |
| go-genai streamed function args | arguments kept per client, not per call | pass | invariant_violated(streamed-call-args)@31 |
| updo policy alerting | policy implemented, alert never sent | pass | invariant_violated(alert-payload)@56 |
```

I didn't expect the first row. In DeepSWE's httpx reference, the function that stores a cookie never runs the limit check, so the store grows past the spec's limits. The hidden tests never set enough cookies to notice. I wrote it up in [The answer key is wrong](https://www.thekavachlabs.com/blog/the-answer-key-has-a-bug).

The other three rows are builds I wrote on purpose, to find what each task's hidden tests miss. They're plausible mistakes, not ones an agent made. The [results and method](https://github.com/kavachlabs/kavach/blob/main/docs/design/implement.md#results-across-all-five-track-b-feature-tasks) list every build, including the wrong ones the hidden tests do catch.

## Agents, with and without it

Does any of this help an agent? The benchmark runs OpenCode with the same prompt in every arm; only the model and Kavach's MCP tools change, and the hidden test grades. The first trials, on four Redis incidents with small models:

```markdown
| | Without Kavach | With Kavach |
| --- | --- | --- |
| Trials that pass the hidden test | 18 / 25 (72%) | 23 / 25 (92%) |
```

That's 25 trials per arm: enough to show the harness works, not enough to claim an effect size. Every run is public at [benchmark.kavachlabs.dev](https://benchmark.kavachlabs.dev).

## What it misses

Where does it get things wrong?

```markdown
**Narrow fixes still get through.** On redis-11734, two wrong fixes (`type-before-args`, `missing-as-empty`) pass `diff`. On redis-10068, a fix that only differs on state saved by an older Redis build passes both Kavach and the hidden test. Array-element variants were added after two narrow Redis fixes slipped past the first version. When something slips through, I add the mutation that would have caught it and rerun.

**One writer at a time.** Kavach assumes a single handler folds the journal into state. Multi-writer services, thread interleavings inside a step, and bugs that span several services are out of scope. That is the territory of [Antithesis](https://antithesis.com), and it's very good at it.

**You have to integrate.** The handler has to read time, randomness, config and gateways through the SDK. For a service already shaped like a consume loop that's a small change; Redis took about 570 lines of C and no change to the code under repair. For a request-handling app with ambient I/O everywhere it's a real refactor. Of the 613 problems I screened, only 30 are service-shaped.

**Fixtures hold production data.** Inputs and gateway responses are recorded as they were. Secrets in the environment are redacted, but field-level redaction at record time isn't built yet, so don't commit a fixture with customer data in it.

**Pre-alpha.** The format and the APIs will change before 1.0.
```

## How it compares

```markdown
**rr and Pernosco** record a process at the instruction level, for a person in a debugger, on Linux, for native code. Kavach records at the handler boundary, in any language, and gives an agent a verdict for each candidate patch.

**VCR-style tapes** stub your dependencies with recorded responses. Kavach does serve the gateway responses your handler read from the fixture, but the point is to rerun *your* code on exactly what it saw, and then on inputs it didn't see.

**Temporal and event sourcing** share the rule at Kavach's core: if state is a fold over a journal, and the fold is deterministic, you can replay it. I [wrote about why that matters](https://www.thekavachlabs.com/blog/event-sourcing). Kavach applies the rule to checking a fix, not to running the service.

**Property-based testing** generates inputs from a generator you write. Kavach's variants start from one real incident, and only the variants that break the old build count.

**A regression test** is what you keep afterwards. Copy the fixture into `testdata/` and `kavach replay` exits 0 once the failure is gone. But a single regression test is exactly the one input an agent overfits to, which is why `diff` exists.
```

## Try it

To see it without installing anything, open [try.kavachlabs.dev](https://try.kavachlabs.dev). It has saved agent runs, with and without Kavach, such as [an agent fixing redis-11734](https://try.kavachlabs.dev/#/jobs/sly-rust-fox-4qxwr98) and replaying the crash on each attempt.

To run it on your own code:

```bash
curl -fsSL https://www.thekavachlabs.com/install.sh | sh
```

```tabs
--- Claude Code | bash
claude mcp add kavach -- kavach mcp
--- Codex | bash
codex mcp add kavach -- kavach mcp
--- OpenCode | json title="opencode.json"
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "kavach": { "type": "local", "command": ["kavach", "mcp"] }
  }
}
```

The install script also adds the four Kavach skills to Claude Code, and to Codex and OpenCode when it finds them. The [README](https://github.com/kavachlabs/kavach#readme) walks through the ledger demo, and there's a Kafka version of it that replays with no broker running.

## What I'm looking for

I spent five years on financial systems where every serious incident was a replay problem nobody could actually run. Kavach is my attempt to make that replay something you can run, and something an agent has to pass before a person reviews its fix.

What it needs now is an incident that isn't mine. Do you run a service that consumes from a queue or a log, and have you had a failure you couldn't reproduce? I'd like to record it with you and see whether the variants find what your tests didn't. Write to [hello@thekavachlabs.com](mailto:hello@thekavachlabs.com), or open an issue on [GitHub](https://github.com/kavachlabs/kavach).

## References

1. [Kavach](https://github.com/kavachlabs/kavach) (Code, github.com)
2. [BENCHMARKS.md](https://github.com/kavachlabs/kavach/blob/main/BENCHMARKS.md) (Code, github.com)
3. [agent guide](https://github.com/kavachlabs/kavach/blob/main/AGENTS.md) (Code, github.com)
4. [results and method](https://github.com/kavachlabs/kavach/blob/main/docs/design/implement.md#results-across-all-five-track-b-feature-tasks) (Code, github.com)
5. [benchmark.kavachlabs.dev](https://benchmark.kavachlabs.dev) (Article, benchmark.kavachlabs.dev)
6. [Antithesis](https://antithesis.com) (Article, antithesis.com)
7. [try.kavachlabs.dev](https://try.kavachlabs.dev) (Article, try.kavachlabs.dev)

Source: https://www.thekavachlabs.com/blog/replay-the-crash-prove-the-fix