How to Read a Kavach Strip Chart

Before I introduce the strip charts on our blog, let me tell you who I am and where this started. I have worked in financial industry for about half-a-decade, and when I recently build HFT infrastructure from scratch, we had to read sequence timelines without getting lost in text logs. While building that, I found certain patterns that are helpful for me, and also helpful for agents, to debug faster, to reproduce the issue, and to verify the fixes.

A journal strip chart¹² is about drawing the events which cause transitions in your state machine. What is a strip chart, you may ask? A strip chart is a software's visual tape which determines what inputs, reads, and outputs took place in each step.

Why does a strip chart spark curiosity and help to debug a bug faster? Whenever we are stuck reproducing what happened by looking at the logs, the network packets, or different containers, a simple strip chart lays out in bare bones the events that took place, which caused the software to end up in the state that it did. Let's say you have a bank account. Your bank account is a summation of all the credits and debits of your money. Now imagine one day you go up to your bank, and it says that you have 10 dollars. When you checked last night, you had 100k. So you will demand the transaction ledger for this, right? You are not ready to accept a guess from a log line when you can inspect the tape.

A Kavach strip chart is kind of like that. It has a transactional ledger which keeps account of all the state changes, all the state mutations that lead up to the crash. In a way, it makes the execution of an application auditable on every step.

Now, since I have worked in the finance industry, I see its applications everywhere. It helps to reproduce bugs, to actually see what events preceded the crash happening, and so on.

Why the shapes inside a step matter#

Starting from a snapshot, if you keep reading the events that are in the journal, each column shows what happened in that step. That's the ultimate goal for our reference diagram³⁴. What makes it interesting are the shapes sprinkled across the lanes. A solid block comes from an input. A thin tick can come from the clock. Four bars can come from the fact that we are reading random bytes before emitting a hollow box.

Having the ability to read deterministically what happened in production from these four lanes is one of the key guarantees of our journal format, and that is what makes the diagram so useful.

func (l *Ledger) Handle(env kavach.Env, in kavach.Input) error {
	var ev Event
	if err := json.Unmarshal(in.Data, &ev); err != nil {
		return fmt.Errorf("decode event at %s: %w", in.Position, err)
	}
	amount := *ev.Amount 
	if amount <= 0 {
		return reject(env, ev, "amount must be positive") 
	}
	at := env.Now() // records 'clock' (0x02) in Lane 2
 
	switch ev.Type {
	case "deposit":
		l.Net += amount
		return l.post(env, ev, ev.Account, amount, at) // 1x (rand + output)
	case "transfer":
		if l.Balances[ev.Account] < amount {
			return reject(env, ev, "insufficient funds") // clock + output (no rand)
		}
		if err := l.post(env, ev, ev.Account, -amount, at); err != nil {
			return err
		}
		return l.post(env, ev.To, amount, at) // 2x (rand + output)
	}
	return nil
}

Challenges of reading production traces#

These challenges all come down to seeing what happened inside a step. Reading a production trace has several such traps.

Two-leg transfers and single deposits. Can a text log show everything? Show every branch? How does it know when a step is a transfer? Which helper ran twice? If these reads aren't drawn on the tape, the same transfer looks just like a deposit in the logs. As a company, Kavach Labs is going to tackle these challenges so that any trace is easy to read.

Early rejections and silent steps. Software often checks inputs in more than one place. Say you design a service that rejects a bad amount before reading the clock. How do you see that without stepping through a debugger? With 1,000 requests it may look fine, but on an early rejection you are skipping the clock and random reads altogether.

The yellow band and red crash marker. One system requirement for a crash fixture is that your failing step should stand out right away. Most single-threaded deterministic software follows this step order, but quite a few log viewers still don't. How do you model those? These are the uncomfortable parts of the system that we want to tackle.

We're building Kavach Labs to tackle this problem head-on. We love deterministic verification as an architectural concept, and we want agentic software to be written and deployed so that it's easy to read a journal tape, replay the crash, fix the code, and deploy the fix with more confidence.