A Rust framework · MIT / Apache-2.0
Deterministic, replicated services on the sequencer architecture — the pattern behind Island/INET, Nasdaq, Jane Street, and LMAX — with deterministic simulation testing built into the runtime.
Navigate with → ← or the tape below · every slide has something to poke at
The idea in sixty seconds
A single logical sequencer imposes a total order on all inputs, stamping each with a gapless u64 sequence number, and appends them to a durable journal. That ordered stream — the tape — is the system of record. Deterministic state machines consume the identical stream and compute bit-identical state.
apply() runsEverything downstream — standby replicas, audit, drop-copy, market data — is just another deterministic consumer of the sequenced stream. That uniformity is the whole point: recovery, replication, debugging, and testing all become operations on the tape.
Rule 1 of 2
apply is a pure function of (state, input). Feed two replicas the identical stream and their states are byte-identical — every replica, every replay, every machine. Try it, then flip the cheat switch.
Both replicas consume the same sequenced inputs.
This is why the framework can ship inputs instead of state, why recovery is replay, and why a failing test seed is a complete reproduction. One ambient read — a clock, a random number, a HashMap iteration order — and all of it collapses. So ticktape doesn’t hope; it enforces (slide 8).
Rule 2 of 2
Wall-clock time enters the system only as sequenced timestamps assigned by the sequencer, monotonically clamped so a stepping clock (NTP, VM migration) can never run sequenced time backwards. Inside apply, Ctx is the only door. Click anything below.
What you actually write
No networking code, no ordering code, no durability code, no recovery code. Hover (or tap) the highlighted regions.
struct Counter { value: i64 } #[derive(Encode, Decode)] enum Cmd { Add(i64), Reset } impl Service for Counter { fn genesis(_: &()) -> Self { Counter { value: 0 } } fn apply(&mut self, seq: Seq, cmd: &Cmd, ctx: &mut Ctx<Evt>) { match cmd { Cmd::Add(n) => self.value += n, Cmd::Reset => self.value = 0 } ctx.emit(Evt::Value(self.value)); } fn snapshot(&self) -> i64 { self.value } fn restore(v: i64, _: &()) -> Self { Counter { value: v } } } let mut node = Node::<Counter>::open(config, ())?; let (seq, out) = node.submit(Cmd::Add(42))?;
One trait: genesis, apply, snapshot, restore. The framework supplies total ordering, durable journaling, replay recovery, replication, and the fault-injecting simulator — for this exact struct, unmodified.
The same struct runs single-node, inside the simulator under crash fault injection, and replicated behind the gateway. That’s the framework’s promise: write the state machine once.
What the tape buys you · 1
In-memory state is a disposable projection of the journal. There is no separate recovery code path to get wrong — recovery runs the same apply that ran live. Build some state, then kill the node.
With snapshots on, restart becomes restore(snapshot) + replay(tail) — same guarantee, less work. The simulator byte-compares the recovered state against an independent full replay after every simulated crash, so your restore is proven exact, not assumed.
The system of record
Every sequenced record is a Frame — fixed little-endian header, opaque app payload, CRC32C over each. Hover the fields. Only inputs are journaled; outputs are recomputable.
A directory of segments named by first seq (…0001.seg), each with a CRC’d header (magic TKTJ, version, first seq, epoch). fsync is a policy: per-frame, time-window group commit, or never — pick your durability tier. Compaction (compact_below) + snapshot pruning bound disk for 24×7 operation.
A torn tail in the final segment is truncated to the last intact frame. Corruption anywhere else is a loud error — never silently-wrong data.
Discipline with a compiler
HashMap/HashSet (iteration order) and bare floats (NaN, −0.0) have no Encode/Decode impls. They cannot appear in inputs or snapshots. This isn’t a lint you can silence — it’s a missing trait bound:
error[E0277]: the trait bound
`HashMap<u64, Order>: Encode` is not satisfied
--> src/lib.rs:12:10
|
12 | #[derive(Encode, Decode)]
| ^^^^^^ use BTreeMap — deterministic order
Ctx is the only doorInside apply: ctx.now(), ctx.seq(), ctx.emit(). Deliberately no spawn, no sleep, no randomness, no file, no socket. External interactions are split-phase: emit a request event; the response re-enters as a sequenced input.
Node::verify_replay() re-runs genesis + replay(journal) and byte-compares snapshots. The simulator performs the same check after every simulated crash. The acceptance suite plants real bug classes — ambient state, overdrafts, an off-by-one restore, injected bit rot — and proves each is caught.
Monotonic clamping at the sequencer means NTP steps and VM migrations can never make sequenced time run backwards.
The reason to trust any of this
Storage and time sit behind traits; ticktape-sim swaps in a simulated disk and virtual clock and runs a whole node in one thread, on one seed — interleaving fsyncs, sequenced ticks, and power-loss crashes where files keep only a seeded prefix of unsynced bytes, possibly with a torn sector. A failing seed is the reproduction.
After every crash+recovery the real harness checks: recovery succeeds · durability (synced frames survive) · total order (a byte-exact gapless prefix) · determinism (recovered state ≡ independent replay — through snapshots too) · your invariants. CI fuzzes 2,000 fresh seeds per push at ~25,000× wall-clock. Seed 0 of the snapshot milestone’s first fuzz run found a real stale-snapshot-poisoning bug — that’s why recovery purges snapshots past the surviving tail.
What the tape buys you · 2
A MoldUDP64-shaped reliable sequenced transport: redundant A/B UDP feeds, heartbeat high-water marks, gap detection by seq, and unicast TCP range retransmission. Followers recompute bit-identical state live. Sabotage it:
Two independent multicast feeds; the follower takes the first copy of each seq it sees.
The reliability core (Reassembler) is a pure state machine, fuzzed across 200 seeds of loss, duplication, and reordering. When both feeds lose a frame, the gap is detected by seq and back-filled from the retransmitter — a bounded in-memory repeater chained to a journal-backed rewinder, so late joiners can catch up from disk without unbounded memory.
What the tape buys you · 3
A lone sequencer can’t be split-brain-safe by itself. Leadership is an epoch lease granted by a majority of acceptors — Paxos-phase-1-shaped, provably at most one leader per epoch — and every failover is fenced with an EpochChange frame on the tape itself.
The leader stamps frames; followers journal and apply the same tape.
Ack after local durability. Fastest; a failover may lose a bounded unreplicated suffix — the historical exchange tradeoff, stated honestly.
VSR-shaped: commit only what a majority has durably journaled. A quorum round-trip buys no committed loss — verified in the cluster sim across leader kills, partitions, zombie leaders, and dueling candidates.
The multi-node simulator drove two real design rules: election winners must reconcile their journal against the quorum’s fences before leading, and replica acks are epoch-scoped — a fenced-off suffix vouches for nothing. Shipping in 1.0: a packaged 3-node deployment with automatic failover — a standby’s failure detector promotes it with no operator action, preserving the exact pre-failover state with zero committed loss.
Where real clients meet the tape
Clients are messy: they time out, retry, and disconnect. The gateway gives each session a monotonic client-seq and journals session envelopes through the sequencer — so dedup state is deterministic, replicated, and survives restarts.
Duplicates are dropped by client-seq — a retry has exactly-once effect.
Also in the box: windowed flow control (window 1 = the classic single-outstanding discipline), gap rejection, cancel-on-disconnect injected as a sequenced input (so every replica pulls the orders identically), drop-copy observers, and a per-session replayable outbox — every event carries a per-session event_seq, so a reconnecting client or observer is backfilled exactly what it missed. The exchange example runs the order book behind this gateway with real TCP clients — and fuzzes session traffic under fault injection.
The workspace
Focused crates — depend only on what you need. Click to expand.
Plus the examples: counter (hello world) · kv (smallest real service, ~90 lines) · orderbook (the flagship price-time-priority CLOB, fuzzed with exchange-grade invariants: never a crossed book, every accepted share exactly one of traded/canceled/resting) · feed (multi-process leader/follower over UDP) · exchange (the book behind the TCP gateway, killed clients and all).
Numbers vs. budgets
ticktape-bench runs in CI (report-only — runner disks are noisy). Bars show measured ÷ budget on a log scale; the tick is the budget line. Linux CI runner numbers:
| path | measured | budget | vs budget (log) |
|---|
The per-frame synced-fsync tail is the honest gap, and it’s architectural: a single serial submit pays a full fsync per frame. The fix — group commit — shipped: Node::submit_batch commits a whole batch with one fdatasync, byte-identical to N submits, which turns ≈ 4 ms/input into ≈ 66 µs/input (~60×) and reaches the budget on real NVMe. Hardware CRC32C (10.7 GB/s), packet batching, and streaming replay round it out; a feature-gated io_uring journal backend goes further on Linux. For pure throughput, fsync=never + Tier 1/2 replication remains the historical exchange configuration.
Status · v1.0.0 · cargo add ticktape
1.0.0 is out and on crates.io. Everything above plus quorum commit (no committed loss), deterministic timers (good-till-date orders), replayable gateway sessions, the performance workstream (group commit, hardware CRC32C, packet batching, streaming replay, feature-gated io_uring + shared-memory ring), a WIRE.md spec, an SBE interop adapter, and a raft-rs delegation backend — dual-platform verified (macOS arm64 + Linux x86_64, io_uring on a real kernel). The public API follows semver from here.
The closest incumbent — JVM-centric, full Raft, no fused simulation harness. ticktape: native Rust, single sequencer, explicit tiers, simulator in the box.
An intra-process ring buffer — a building block, not a durable replicated system. The vocabulary (and split-phase discipline) comes from there.
Sets the bar for simulation rigor (VOPR) — as a fixed accounting state machine in Zig. ticktape aims that rigor at arbitrary services in Rust.
Consensus libraries: they order a log but bring no deterministic runtime, codec discipline, or service ergonomics.
A different problem: ms-class brokered messaging, per-partition order, non-deterministic consumers.
One logical sequencer caps throughput (sharding by stream_id is the named mitigation). Acceptor durability is on the embedder. Recovery still CRC-reads the full journal.
git clone https://github.com/matthart1983/ticktape
cd ticktape && cargo test --workspace # incl. seeded fuzz
cargo run --release -p ticktape-sim --bin vopr
github.com/matthart1983/ticktape
MIT OR Apache-2.0 · Rust stable
Background: Fowler’s The LMAX Architecture · Nigito’s How to Build an Exchange