81.2 Nanoseconds: Designing an Always-On Production Flight Recorder

Before I introduce our flight recorder, let me tell you who I am and where this started. I have worked in financial industry for about half-a-decade, and when I recently build HFT infrastructure from scratch, we had to record live traffic without slowing down the hot path. While building that, I found certain patterns that are helpful for me, and also helpful for agents, to debug faster, to reproduce the issue exactly, and to verify the fixes.

Flight recording¹² is about journaling the inputs which cause transitions in your service handler. What is a determinism boundary, you may ask? A determinism boundary is a software's internal line which determines what events get captured for offline replay.

Why does a lightweight boundary help to debug a bug faster? Whenever we are stuck reproducing what might have happened by looking at the logs, the network packets, or different containers, a simple handler boundary lays out in bare bones the events that took place, which caused the software to end up in the state that it did. Let's say you have a bank account. Your bank account is a summation of all the credits and debits of your money. Now imagine one day you go up to your bank, and it says that you have 10 dollars. When you checked last night, you had 100k. So you will demand the transaction ledger for this, right? You are not ready to accept a guess from a test lab when your live balance is wrong.

An in-process flight recorder is kind of like that. It has a transactional ledger which keeps account of all the state changes, all the state mutations that lead up to the crash. In a way, it makes the execution of an application auditable on every step.

Now, since I have worked in the finance industry, I see its applications everywhere. It helps to reproduce bugs, to actually see what events preceded the crash happening, and so on.

func (l *Ledger) Handle(env kavach.Env, in kavach.Input) error {
    now := env.Now()          // recorded as a 'clock' frame
    var nonce [8]byte
    _, _ = env.Read(nonce[:]) // recorded as a 'rand' frame
 
    entry, err := l.apply(in.Data, now, nonce)
    if err != nil {
        return err
    }
    env.Emit("ledger.entries", entry) // recorded as an 'output' frame
    return nil
}

Why shared memory matters for recording#

Starting from a snapshot, if you keep replaying the events that are in the journal, you should end up with the exact crash state. That's the ultimate goal for a flight recorder. What makes it challenging are certain bits of overhead sprinkled through the operating system. Overhead comes from the kernel. It can come from system calls. It can come from the fact that we are copying bytes over a pipe on every event.

Having the ability to record deterministically in shared memory in about eighty nanoseconds is one of the key guarantees of Kavach³⁴, and that is what makes always-on recording possible in production.

go test -run '^$' -bench RecorderStep -benchtime 2s -count 5 ./replay

Challenges of recording live production services#

These challenges all come down to recording without slowing down the service. Production recording has several such traps.

Losing the last event on a crash. Can a flight recorder capture everything? Capture every panic? How does it know what the input was when the process died? Which step was the handler running? If these events aren't in shared memory, a crash wipes out the very input you needed to see. As a company, Kavach Labs is going to tackle these challenges so that the shared ring keeps the last input safe on disk.

Slowing down the hot path. Software often writes logs or traces in more than one place. Say you design a service that writes every step over a pipe. How do you record without paying for a system call on every event? With 1,000 requests a second it may behave fine, but under a traffic spike you are blocking the handler on the kernel.

Replaying without external services. One system requirement for fast replayability is that your handler should read its clock and inputs from the journal. Most single-threaded deterministic software follows this principle, but quite a few chunks of software still don't. How do you model those? These are the uncomfortable parts of the system that we want to tackle.

We're building Kavach Labs to tackle this problem head-on. We love deterministic recording as an architectural concept, and we want agentic software to be written and deployed so that it's easy to capture a bug, replay it, fix the code, and deploy the fix with more confidence.