The Hidden Tests Pass. The Contract Doesn't.

Before I introduce feature contracts, let me tell you who I am and why we built this. I have worked in financial industry for about half-a-decade, and when I recently build HFT infrastructure from scratch, we never shipped a new order type without replaying recorded traffic against it first. While building that, I found certain patterns that are helpful for me, and also helpful for agents, to debug faster, to reproduce the issue exactly, and to verify the fixes.

Contract verification¹² is about locking the traffic and invariants which govern transitions in your state machine. What is a behaviour contract, you may ask? A behaviour contract is a software's locked baseline which determines whether old traffic stays untouched and new feature traffic obeys the rules.

Why does a behaviour contract help to catch regressions faster? Whenever we are stuck reviewing a feature pull request by looking at unit tests that the agent wrote itself, a simple contract check lays out in bare bones the events that took place, which caused the software to change its output. Let's say you have a bank account. Your bank account is a summation of all the credits and debits of your money. Now imagine one day you go up to your bank, and it says that you have 10 dollars. When you checked last night, you had 100k. So you will demand the transaction ledger for this, right? You are not ready to accept a green unit test when a new fee rule silently changed existing balances.

A feature contract is kind of like that. It has a transactional ledger which keeps account of all the state changes, all the state mutations that lead up to the new feature's output. In a way, it makes the impact of a new feature auditable.

Now, since I have worked in the finance industry, I see its applications everywhere. It helps to catch silent regressions, to actually see what existing events changed their output, and so on.

kavach contract init \
  --spec SPEC.md \
  --traffic traffic.kavach \
  --feature-match '$.argv[0]=="NEWCMD"' \
  --invariant capacity_non_negative \
  --invariant-source handler.go
 
kavach contract check --old /tmp/old --new /tmp/old

Why locking the contract matters#

Starting from a snapshot, if you keep replaying the events that are in the journal, old steps should match the old build and new steps should satisfy the invariants. That's the ultimate goal for contract verification³⁴. What makes it challenging are certain habits sprinkled through coding agents. Regressions come from shared helpers. They can come from weakened checks. They can come from the fact that the agent is editing the test to make it pass.

Having a human approve and lock the contract before the agent writes the code is one of the key guarantees of our workflow, and that is what makes the verification trustworthy.

Challenges of evaluating coding agents on features#

These challenges all come down to what happens outside the happy path. Evaluating an agent's feature has several such traps.

Silent regressions on existing traffic. Can a unit test suite catch everything? Catch every changed byte? How does it know what the old output was on a thousand earlier requests? Which helper function did the agent touch? If these old requests aren't replayed, the same events can produce different outputs in production. As a company, Kavach Labs is going to tackle these challenges so that any regression on existing traffic is caught right away.

Agents editing the checks. Software often keeps tests and implementation in the same repository. Say you ask an agent to implement a feature that fails an invariant check. How do you verify the code if the agent can edit the invariant file itself? With 1,000 lines of diff it may look fine, but in the middle the agent weakened the rule to make the test pass.

Edge cases that happy-path tests miss. One system requirement for feature verification is that your invariants should hold across variations of the new command. Most careful engineers test these edge cases, but quite a few chunks of agent code still don't. How do you model those? These are the uncomfortable parts of the system that we want to tackle.

We're building Kavach Labs to tackle this problem head-on. We love contract verification as an architectural concept, and we want agentic software to be written and deployed so that it's easy to lock a contract, replay it, fix the code, and deploy the feature with more confidence.