The answer key is wrong, the agent got it right, and the benchmark scores them the same

Lately I have been running coding agents against a benchmark called DeepSWE¹. What is DeepSWE? A public set of feature requests from real repositories, where the agent writes the code and hidden tests, which the agent never sees, decide whether the feature is done. I build a tool, Kavach, which records the traffic of a service and replays it through another build, and one thing which kept bothering me was how much those hidden tests see. So I pointed the replay tool at the graders themselves, and httpx is where it got interesting.

The httpx task asks for a new cookie store, and the feature request has one paragraph about limits, pasted as it is:

Every task ships with a reference solution, written by the task authors so they know their tests can be passed. At some point I did a simple thing with it: a limit of two, and three cookies. What happens?

$ python -c '
import httpx
store = httpx.CookieStore(max_cookies=2)
store.set("a", "1"); store.set("b", "2"); store.set("c", "3")
print(len(store), sorted(store))'
3 ['a', 'b', 'c']

It keeps three. The reference calls the limit check after it extracts cookies from a response, and inside update, but set, and item assignment also, go through a separate internal function which doesn't call it. I didn't find this by reading the code. My check failed on the reference at step 190, saying the store holds 13 cookies while the limit is 12, and for a good while I assumed my invariant was wrong, since the answer key cannot be wrong. Then I read the patch.

Why do the hidden tests not catch this? There are 115 of them, three are about eviction, and all three add cookies by extracting them from a response. None of them call set on a store which has a limit, and that's the entire gap. Add one call to the limit check in that function and the reference passes my check, and still all 115 hidden tests.

The part which actually bothers me#

I have one real agent run on this task, from October 14: GLM-5.3 through OpenCode, one attempt in each of my four arms, and the arm I care about is the control arm, with nothing from Kavach. Did the agent make the same mistake? It didn't. It enforced the limits inside set, passed the hidden tests 115 out of 115, and passed my contract check also, 59 of 59 feature steps changed, 64 of 64 variants, and so did the other three arms. So the reference has a reward of 1, the agent has a reward of 1, and the benchmark can't tell an answer key with a bug apart from a model which didn't have the bug. If somebody is training on that reward, what is it teaching the model? It's one model and one trial per arm, so I'm not claiming any trend, only that the grader has a blind spot and I can point at it.

How did I get here?#

The plan was kind of simple: take the five feature tasks in DeepSWE, write wrong implementations for each by hand, mistakes which an agent in a hurry makes, and grade every wrong build in two ways, with the hidden tests and with a Kavach contract. What is a contract? Traffic recorded before the feature, plus a few invariants which a person has read and approved, details in the box below. For httpx the limits invariant is a single sentence, the store never holds more than its limit, and the traffic opens a client with a limit of twelve and later sets a thirteenth cookie.

$ kavach contract check --dir contract \
    --old "python /opt/httpx-old/kavach_driver.py" --new "python kavach_driver.py"
baseline  ok
new       invariant_violated(cookie-limits)@190
features  33 of 59 feature steps changed
verdict   invariant_violated(cookie-limits)@190
          client shop: holds 13 cookies, max_cookies=12
wall      223.3 ms

Nine wrong builds, three of them pass the hidden tests, all nine fail the contract check, the reference fails it too, and the other four references pass.

Where I would push back on myself#

Is this independent evidence? Not really, since I wrote the wrong builds and the invariants also, and five tasks is few. You could argue "when limits are exceeded" only means during extraction; I don't read it that way, but someone will. And Kavach isn't free, the httpx driver is about 470 lines, and it only works on single-writer services. Does any of this matter if nobody trains on DeepSWE rewards? I think people do, but that's for them to say.

Reproduce it#

One script, no Docker, you need Go, Python 3.12 and the network. Kavach is Apache 2.0 and pre-alpha. If you have a feature request and the build before it, I'd like to write the contract with you and see what your tests miss. Write to hello@thekavachlabs.com.

git clone https://github.com/kavachlabs/kavach && cd kavach
bench/agent/tasks/httpx-deterministic-cookie-store/run.sh