I graded 26 fixes to six SWE-bench tasks two ways. The graders disagreed on four.
Six incident tasks from SWE-bench Multilingual, 26 hand-written candidate fixes, each graded by the upstream hidden test and by Kavach replaying the recorded incident. They disagree on four, two in each direction, and a real agent run fell into one of Kavach's holes.
This is the companion to the answer-key post, about the incident tasks. Six come from SWE-bench Multilingual: four from Redis, one from Valkey and one from axum. For each one I recorded an incident on the old build, the traffic of a small service built around the bug report, wrote 26 candidate fixes by hand, correct, narrow and wrong ones, and graded each in two ways: the upstream hidden test, and Kavach replaying the incident against the fixed build. What does Kavach check? That the recorded failure is gone, that every step before it still produces the same bytes, and that variants of the incident which break the old build pass on the new one; details in the box below.
Deep diveWhat kavach diff does with an incident
It replays the fixture on the old build (expecting the recorded failure) and on the new one. On the new build every step before the failure must produce the recorded outputs (diverged@N otherwise), the failing step must not fail, and declared invariants must hold. It then makes variants of the incident (fields of the failing input changed or removed, array elements swapped with other inputs', the input moved earlier, earlier inputs dropped or redelivered, the clock shifted), keeps those that make the old build fail the same way, and requires the new build to pass all of them (variant_failed(K)@N). Regression variants apply the same mutations to earlier inputs the old build handled; the new build must match its outputs. Fewer than 10 reproducing variants gives unverified, not fixed.
So how often do two graders agree on 26 fixes? On 22 of them. The four where they disagree go two each way.
| Task | Candidate | Hidden test | kavach diff |
|---|---|---|---|
| axum-1934 | skip-matched-path: never set MatchedPath on any route | pass | diverged@1 (metrics label /api/todos became unmatched) |
| valkey-1499 | scripts-always-touch: drop no-touch mode while a script runs | pass | invariant_violated(access-time-contract)@57 |
| redis-11734 | type-before-args: WRONGTYPE before the argument errors | fail | fixed |
| redis-11734 | missing-as-empty: a missing key is an empty string | fail | fixed |
Take axum first. The hidden tests check that a fallback router answers without panicking, which that candidate does. It also stops setting the matched path on every route, which breaks the recorded service's metrics middleware, and Kavach sees it on the first request, where the label changes from the API route to unmatched, long before the panic. Valkey has the same shape: the candidate makes TOUCH work by dropping no-touch mode for everything a script reads, and the recorded cache keeper peeks at a value from a script at seq 57, after which an idle time of 3 seconds in production comes back as 0. Why doesn't the hidden test see it? It never reads a key from a script in no-touch mode. What Kavach has here is the behaviour before the failure, which a test written for the failure doesn't have.
Now the other direction, where Kavach is wrong. Two wrong BITPOS fixes in redis-11734 pass Kavach and fail the hidden test. One keeps the type check ahead of the argument parsing, so a list key answers WRONGTYPE where the arguments are invalid, and the fixture holds only string keys, so no variant reaches it. The other treats a missing key as an empty string, so a missing key asked for a clear bit replies minus one, not zero, and the recorded dashboard only ever looked for set bits. Those are Kavach misses. Could I change the traffic to catch them? Yes, and that would be tuning the fixture to the candidates, so I didn't.
Did a real agent fall into the same hole? One did. On October 15, a Qwen flash model in the Kavach arm finished with Kavach saying fixed, 46 of 46 variants passing, and the hidden test failing eleven BITPOS tests. Its edit duplicated the range-conversion block, so bit-offset intervals came out wrong, and the recorded traffic only ever asked for the whole bitmap or one byte range, so neither the fixture nor its variants asked for a bit interval. Kavach told the agent it was done, and it wasn't.
# bench/agent/results/harbor/hardy-teal-marmot-1u95k69/redis-11734-kavach__G3q9V5M
verifier/grade.json
hidden fail
kavach_diff fixed
variants 46 of 64 reproduce the incident on the old build
46 of those pass on the new build
verifier/grade.log (11 of 49 tests in unit/bitops fail, among them)
[err]: BITPOS bit=0 starting at unaligned address
[err]: BITPOS bit=1 works with intervals
[err]: BITPOS/BITCOUNT fuzzy testing using SETBITThen there are five candidates both graders accept which I wouldn't merge. What does each one miss? The table says. Kavach doesn't beat the hidden test on any of them, and for anyone deciding whether to trust it, that's the row which matters.
| Task | Candidate | Hidden test | kavach diff | What neither sees |
|---|---|---|---|---|
| redis-10068 | split-node-on-big-delta | pass | fixed | a stream saved by the old build still loses every entry to XTRIM MINID |
| redis-13115 | hincrby-accepts-float | pass | fixed | INCRBY, EXPIRE and SET from a script still get 2e+8 |
| redis-13115 | snprintf-17g | pass | fixed | 0.1 stored as 0.10000000000000001; 1e17 still rejected |
| valkey-1499 | eval-only | pass | fixed | EVALSHA, FCALL and the _RO forms |
| axum-1934 | remove-stale-matched-path | pass | fixed | the fallback's /*__private__axum_fallback route leaks as MatchedPath |
What did the agents do with it?#
| Task | Without Kavach | With Kavach |
|---|---|---|
| redis-10068 | 1/1 | 1/1 |
| redis-10764 | 2/2 | 2/2 |
| redis-11734 | 7/12 | 11/12 |
| redis-13115 | 9/10 | 10/10 |
| Total (hidden test passes) | 19/25 | 24/25 |
Models: glm-5.3, glm-5.3-flash, glm-flash-latest, qwen3.8-flash, pareto-latest, gemini-3.8-flash, deepseek-flash-latest, ling-3.1-flash, mimo-v2.6-flash; one or two trials per model and task. The axum trials errored and are not counted. On redis-11734, pareto-latest: 0/4 without Kavach, 4/4 with it.
Four Redis tasks, 25 trials in each arm, small flash-tier models. The spread is almost all redis-11734, where the report is stale and the bug sits in a sibling command. The one failure in the Kavach arm is the Qwen trial above. Is 25 against 25 enough to claim an effect? No, and I'm not claiming one, only that the harness works and the numbers will change as runs land.
Why two oracles?#
Both graders are imperfect, and they fail in different places: the hidden test on behaviour it wasn't written to check, Kavach on inputs the traffic never contained. That's the argument for having both, and it's also why the feature contracts in the other post exist. An incident gives you one failing input to vary; a feature gives you none, so the contract records traffic before the feature and a person writes the invariants.
I wrote the candidates, the traffic and the invariants, so none of this is independent evidence, and six tasks is few. The integrations ran from about 220 lines of Rust for axum to about 890 lines of C for Redis blocking commands, and Kavach only covers single-writer services. Every task directory has a script which reproduces its table on a laptop.