# The answer key is wrong, the agent got it right, and the benchmark scores them the same

> DeepSWE's reference solution for the httpx cookie-store task holds more cookies than its own limit. A GLM-5.3 run with no help got that part right. The hidden tests give both a reward of 1, which is the part that bothers me.

Koustav Chowdhury · Founder · October 27, 2026 · Engineering

Lately I have been running coding agents against a benchmark called [DeepSWE](https://github.com/datacurve-ai/deep-swe). What is DeepSWE? A public set of feature requests from real repositories, where the agent writes the code and hidden tests, which the agent never sees, decide whether the feature is done. I build a tool, Kavach, which records the traffic of a service and replays it through another build, and one thing which kept bothering me was how much those hidden tests see. So I pointed the replay tool at the graders themselves, and httpx is where it got interesting.

The httpx task asks for a new cookie store, and the feature request has one paragraph about limits, pasted as it is:

```markdown
**DeepSWE, httpx-deterministic-cookie-store, instruction.md**

> CookieStore must accept optional limits max_cookies and max_cookies_per_domain (ints or None). [...] When limits are exceeded, evict deterministically by oldest creation order, first for the per-domain limit and then for the global limit.
```

Every task ships with a reference solution, written by the task authors so they know their tests can be passed. At some point I did a simple thing with it: a limit of two, and three cookies. What happens?

```bash
$ python -c '
import httpx
store = httpx.CookieStore(max_cookies=2)
store.set("a", "1"); store.set("b", "2"); store.set("c", "3")
print(len(store), sorted(store))'
3 ['a', 'b', 'c']
```

It keeps three. The reference calls the limit check after it extracts cookies from a response, and inside update, but set, and item assignment also, go through a separate internal function which doesn't call it. I didn't find this by reading the code. My check failed on the reference at step 190, saying the store holds 13 cookies while the limit is 12, and for a good while I assumed my invariant was wrong, since the answer key cannot be wrong. Then I read the patch.

Why do the hidden tests not catch this? There are 115 of them, three are about eviction, and all three add cookies by extracting them from a response. None of them call set on a store which has a limit, and that's the entire gap. Add one call to the limit check in that function and the reference passes my check, and still all 115 hidden tests.

## The part which actually bothers me

I have one real agent run on this task, from October 14: GLM-5.3 through OpenCode, one attempt in each of my four arms, and the arm I care about is the control arm, with nothing from Kavach. Did the agent make the same mistake? It didn't. It enforced the limits inside set, passed the hidden tests 115 out of 115, and passed my contract check also, 59 of 59 feature steps changed, 64 of 64 variants, and so did the other three arms. So the reference has a reward of 1, the agent has a reward of 1, and the benchmark can't tell an answer key with a bug apart from a model which didn't have the bug. If somebody is training on that reward, what is it teaching the model? It's one model and one trial per arm, so I'm not claiming any trend, only that the grader has a blind spot and I can point at it.

## How did I get here?

The plan was kind of simple: take the five feature tasks in DeepSWE, write wrong implementations for each by hand, mistakes which an agent in a hurry makes, and grade every wrong build in two ways, with the hidden tests and with a Kavach contract. What is a contract? Traffic recorded before the feature, plus a few invariants which a person has read and approved, details in the box below. For httpx the limits invariant is a single sentence, the store never holds more than its limit, and the traffic opens a client with a limit of twelve and later sets a thirteenth cookie.

```callout
deep-dive | What a contract check does
The traffic is recorded on the build from before the feature, and it includes the inputs that will reach the feature. The invariants are written from the feature request as plain code on the client's state (for httpx: `cookie-scope`, `cookie-order`, `cookie-limits`). A person approves the contract, which locks the traffic, the spec and the invariant source. `kavach contract check` then replays the traffic on the new build and reports `regression@N` if a non-feature step produced different bytes, `unreached@N` if a feature step still produced the old build's output, `invariant_violated(X)@N` if an invariant broke after any step, and `variant_failed(K)@N` if one of 64 variants of the feature steps failed or broke an invariant. Only `pass` passes.
```

```bash
$ kavach contract check --dir contract \
    --old "python /opt/httpx-old/kavach_driver.py" --new "python kavach_driver.py"
baseline  ok
new       invariant_violated(cookie-limits)@190
features  33 of 59 feature steps changed
verdict   invariant_violated(cookie-limits)@190
          client shop: holds 13 cookies, max_cookies=12
wall      223.3 ms
```

Nine wrong builds, three of them pass the hidden tests, all nine fail the contract check, the reference fails it too, and the other four references pass.

```markdown
| Task | Build | Hidden tests | kavach contract check |
| --- | --- | --- | --- |
| httpx cookie store | DeepSWE reference | pass (115/115) | invariant_violated(cookie-limits)@190 |
| httpx cookie store | reference + one line | pass (115/115) | pass |
| httpx cookie store | GLM-5.3, control arm, 2026-10-14 | pass (115/115) | pass |
| httpx cookie store | one store shared by every client | pass (115/115) | regression@13 |
| httpx cookie store | no host-only, Secure ignored | fail (112/115) | invariant_violated(cookie-scope)@112 |
| go-genai streamed args | call state kept per client, not per stream | pass (6/6) | invariant_violated(streamed-call-args)@31 |
| updo alerting | policy built, alert never sent | pass (17/17) | invariant_violated(alert-payload)@56 |
| updo alerting | cooldown suppresses recoveries | fail (14/17) | invariant_violated(alert-delivery)@130 |
| kombu single active consumer | cancel callback errors propagate | fail (p2p 1420/1421) | invariant_violated(cancel-callback-errors-contained)@265 |
| kombu single active consumer | delivery still to last callback | fail (81/85) | unreached@137 |
| kombu dead lettering | deleted queue keeps its properties | fail (74/76) | regression@661 |
| kombu dead lettering | x-death appended, not counted | fail (75/76) | invariant_violated(dead-letter-headers)@499 |
```

## Where I would push back on myself

Is this independent evidence? Not really, since I wrote the wrong builds and the invariants also, and five tasks is few. You could argue "when limits are exceeded" only means during extraction; I don't read it that way, but someone will. And Kavach isn't free, the httpx driver is about 470 lines, and it only works on single-writer services. Does any of this matter if nobody trains on DeepSWE rewards? I think people do, but that's for them to say.

```callout
note | Has anyone reported this?
In October, Tokenless audited this same task and found five requirements its tests never check, such as item assignment and per-request `cookies=` ([their write-up](https://github.com/tokenless-ai/envcheck-findings/blob/post/2026-10-grader-defects/posts/2026-10-grader-defects/findings/httpx-cookie-store-coverage-gaps/README.md), indexed in [DeepSWE issue #105](https://github.com/datacurve-ai/deep-swe/issues/105)). They used the reference solution as the correct baseline, so the missing limit check isn't on their list.
```

```callout
note | The green that worried me more
This isn't the strangest reward 1 in DeepSWE this month. In the same audit, Tokenless showed that on `adaptix-name-mapping-aliases` a patch that changes no product code, only `tests/conftest.py` so that failed pytest reports are rewritten as passed, gets reward 1 with 44/44 feature tests and 2,738/2,738 regression tests passing ([issue #105](https://github.com/datacurve-ai/deep-swe/issues/105)). A grader that runs the submission's own test hooks will believe whatever they say.
```

```callout
note | The incident tasks, graded the same two ways
The other half of the benchmark is six incident tasks from SWE-bench Multilingual, with 26 hand-written fixes graded by the hidden test and by `kavach diff`. The graders disagree on four, two in each direction: [two graders, four disagreements](https://www.thekavachlabs.com/blog/two-graders-four-disagreements).
```

![Kavach on a Go ledger: a null amount crashes the service and the recorder writes a fixture; kavach diff rejects a deposit-only null check because 8 of 47 variants of the incident still panic, then accepts the fix that checks every event type](https://www.thekavachlabs.com/blog/kavach-at-work/kavach-at-work.gif "kavach replay and kavach diff on the ledger example, every command run for real")

## Reproduce it

One script, no Docker, you need Go, Python 3.12 and the network. Kavach is Apache 2.0 and pre-alpha. If you have a feature request and the build before it, I'd like to write the contract with you and see what your tests miss. Write to [hello@thekavachlabs.com](mailto:hello@thekavachlabs.com).

```bash
git clone https://github.com/kavachlabs/kavach && cd kavach
bench/agent/tasks/httpx-deterministic-cookie-store/run.sh
```

## References

1. [DeepSWE](https://github.com/datacurve-ai/deep-swe) (Code, github.com)
2. [their write-up](https://github.com/tokenless-ai/envcheck-findings/blob/post/2026-10-grader-defects/posts/2026-10-grader-defects/findings/httpx-cookie-store-coverage-gaps/README.md) (Code, github.com)
3. [DeepSWE issue #105](https://github.com/datacurve-ai/deep-swe/issues/105) (Code, github.com)

Source: https://www.thekavachlabs.com/blog/the-answer-key-has-a-bug