# 81.2 Nanoseconds: Designing an Always-On Production Flight Recorder

> This post is about where the determinism boundary should live in a backend service, and how our shared-memory ring buffer records every event in 81.2 nanoseconds.

Koustav Chowdhury · Founder · October 14, 2026 · Engineering

Before I introduce our flight recorder, let me tell you who I am and where this started. I have worked in financial industry for about half-a-decade, and when I recently build HFT infrastructure from scratch, we had to record live traffic without slowing down the hot path. While building that, I found certain patterns that are helpful for me, and also helpful for agents, to debug faster, to reproduce the issue exactly, and to verify the fixes.

[Flight recording](https://antithesis.com/blog/deterministic_hypervisor/)[†](https://antithesis.com/blog/2026/retrologging/) is about journaling the inputs which cause transitions in your service handler. What is a determinism boundary, you may ask? A determinism boundary is a software's internal line which determines what events get captured for offline replay.

Why does a lightweight boundary help to debug a bug faster? Whenever we are stuck reproducing what might have happened by looking at the logs, the network packets, or different containers, a simple handler boundary lays out in bare bones the events that took place, which caused the software to end up in the state that it did. Let's say you have a bank account. Your bank account is a summation of all the credits and debits of your money. Now imagine one day you go up to your bank, and it says that you have 10 dollars. When you checked last night, you had 100k. So you will demand the transaction ledger for this, right? You are not ready to accept a guess from a test lab when your live balance is wrong.

![Where should the determinism boundary live? The top lane shows a Full-VM Hypervisor Boundary using x86 instruction-retired PMC and a Guest OS pinned to 1 core per VM, passing through a Container Topology of Broker, DB, and Mocks, where adding a printf causes a Timing Butterfly Effect with no local ARM64 support. The bottom lane shows the Kavach Handler Boundary (SPEC.md §10.7) where a Service Handler Step reads env.Now(), env.Read(), and env.Emit() across an 8 MiB Shared Ring at 81.2 ns per event with 0 syscalls, drained by kavach-recorder into a 1,954-byte .kavach fixture that replays in 61.1 microseconds.](https://www.thekavachlabs.com/blog/81-nanoseconds-in-process-flight-recorder/d-ring-boundary.diagram.svg "Determinizing at the handler environment boundary avoids the x86 instruction-counter butterfly effect")

An in-process flight recorder is kind of like that. It has a transactional ledger which keeps account of all the state changes, all the state mutations that lead up to the crash. In a way, it makes the execution of an application auditable on every step.

Now, since I have worked in the finance industry, I see its applications everywhere. It helps to reproduce bugs, to actually see what events preceded the crash happening, and so on.

```callout
spike-pit | Why syscall recording (rr) isn't free in production either
Tools like Mozilla's `rr` move the boundary up from the hypervisor to Linux kernel syscalls and `perf` counters. That works well for local C++ debugging, but `ptrace` traps, `seccomp` filters, and single-threaded serialization add 1.2x to 2x runtime overhead—far too heavy to leave running inside a live payment ledger or order gateway.
```

```go
func (l *Ledger) Handle(env kavach.Env, in kavach.Input) error {
    now := env.Now()          // recorded as a 'clock' frame
    var nonce [8]byte
    _, _ = env.Read(nonce[:]) // recorded as a 'rand' frame

    entry, err := l.apply(in.Data, now, nonce)
    if err != nil {
        return err
    }
    env.Emit("ledger.entries", entry) // recorded as an 'output' frame
    return nil
}
```

## Why shared memory matters for recording

Starting from a snapshot, if you keep replaying the events that are in the journal, you should end up with the exact crash state. That's the ultimate goal for a flight recorder. What makes it challenging are certain bits of overhead sprinkled through the operating system. Overhead comes from the kernel. It can come from system calls. It can come from the fact that we are copying bytes over a pipe on every event.

Having the ability to record deterministically in shared memory in about eighty nanoseconds is one of the key guarantees of [Kavach](https://github.com/kavachlabs/kavach/blob/main/SPEC.md)[†](https://github.com/kavachlabs/kavach/blob/main/BENCHMARKS.md), and that is what makes always-on recording possible in production.

```bash
go test -run '^$' -bench RecorderStep -benchtime 2s -count 5 ./replay
```

```callout
deep-dive | What the 81.2 ns/event benchmark includes (and doesn't)
The benchmark runs a sustained stream that fills the 8 MiB ring within milliseconds, so the 81.2 ns/event median includes any backpressure wait for `kavach-recorder` to drain, number, and compress frames on another core. It injects a seeded `math/rand` source rather than `crypto/rand` so the benchmark measures Kavach's ring and framing overhead rather than the kernel's CSPRNG.
```

## Challenges of recording live production services

These challenges all come down to recording without slowing down the service. Production recording has several such traps.

**Losing the last event on a crash.** Can a flight recorder capture everything? Capture every panic? How does it know what the input was when the process died? Which step was the handler running? If these events aren't in shared memory, a crash wipes out the very input you needed to see. As a company, Kavach Labs is going to tackle these challenges so that the shared ring keeps the last input safe on disk.

**Slowing down the hot path.** Software often writes logs or traces in more than one place. Say you design a service that writes every step over a pipe. How do you record without paying for a system call on every event? With 1,000 requests a second it may behave fine, but under a traffic spike you are blocking the handler on the kernel.

**Replaying without external services.** One system requirement for fast replayability is that your handler should read its clock and inputs from the journal. Most single-threaded deterministic software follows this principle, but quite a few chunks of software still don't. How do you model those? These are the uncomfortable parts of the system that we want to tackle.

```interactive
/interactive/replay-lab.html?scene=crash
The 33-record ledger crash fixture: scrub through the recording to inspect inputs, clock reads, random reads, and outputs.
```

We're building Kavach Labs to tackle this problem head-on. We love deterministic recording as an architectural concept, and we want agentic software to be written and deployed so that it's easy to capture a bug, replay it, fix the code, and deploy the fix with more confidence.

## References

1. [So you think you want to write a deterministic hypervisor?](https://antithesis.com/blog/deterministic_hypervisor/) (Article, antithesis.com)
2. [Free log storage with a time machine](https://antithesis.com/blog/2026/retrologging/) (Article, antithesis.com)
3. [Kavach Journal and Protocol Specification (v0)](https://github.com/kavachlabs/kavach/blob/main/SPEC.md) (Spec, github.com)
4. [Kavach Reproducible Benchmarks](https://github.com/kavachlabs/kavach/blob/main/BENCHMARKS.md) (Code, github.com)

Source: https://www.thekavachlabs.com/blog/81-nanoseconds-in-process-flight-recorder