netwatch ~ ~/writing/building-software-markets-can-trust.md
writing
netwatchlabs
cd ..
· Updated ·8 min readcapital-marketsreliabilityarchitecture

Building Software That Markets Can Trust

Correctness, replay and failure behaviour in pricing systems

A pricing system can return a number quickly, consistently and incorrectly. It can also calculate the right value and deliver it too late, attach the wrong status, or leave nobody able to explain which inputs it used. Trust depends on the whole path from observation to decision to delivery.

That makes pricing software a useful test of engineering discipline. The calculation needs a defensible model and suitable inputs. The surrounding system needs explicit timing, controlled changes and a record of what happened. Reliability is weakened when either half is treated as somebody else's problem.

Separate the questions

There are at least four questions to answer about a result.

Was the calculation implemented correctly? Does it follow the intended algorithm, conventions and numerical tolerances?

Was the model appropriate? Are its assumptions and limitations acceptable for this instrument and use?

Were the inputs suitable? Were the market data, reference data and configuration valid for the valuation being requested?

Was the result used correctly? Did it reach the intended consumer in time, with its provenance and status intact?

Reproducibility helps investigate all four. It does not answer them by itself. A deterministic program with an incorrect unit conversion can produce the same wrong result indefinitely.

The distinction has a formal counterpart in model-risk practice. The US banking agencies' revised 2026 guidance describes validation in terms of model reliability, limitations, conceptual soundness and outcomes analysis.[1] That is useful context for pricing engineering, though its supervisory scope should not be confused with a universal requirement for every financial system.

A disputed price

Consider an illustrative fixed-income valuation, simplified to isolate the investigation.

An instrument pays a single 1,000-unit cash flow at a specified future date. The pricing calculation multiplies that cash flow by the applicable discount factor. A recorded factor of 0.970 produces 970. A later run uses 0.968 and produces 968.

Replaying the original inputs confirms that the old build returned 970. That establishes what the software did. It does not establish that 0.970 was the right factor to use.

The next check is independent of the replay: inspect the intended curve selection, its valuation time, the market-data inputs used to construct it and the instrument's reference data. Suppose the request required a newly approved curve snapshot with factor 0.968, but a cache supplied the preceding snapshot. Both multiplications are correct. The selection of data made the original result unsuitable for that request.

For this deliberately simple case, a separate calculation verifies the arithmetic. A real instrument may require independently derived reference values, model review, sensitivity checks and analysis of conventions. A second implementation that shares the same mistaken assumptions is a weaker check than it first appears.

The investigation should end with a narrower claim than “replay proves the price.” It shows which path produced the value, while validation establishes whether that path was acceptable.

FIGURE 2.1Reconstruction is not validation
Reconstruction is not validationTwo parallel calculations use a cash flow of 1,000. The recorded factor gives 970; the required factor gives 968. Replay explains the first value; independent input validation challenges its suitability. A / RECONSTRUCT THE RECORDB / CHECK THE REQUIREMENTRecorded curve snapshotdiscount factor 0.970Required curve snapshotdiscount factor 0.9681,000 × 0.970 = 9701,000 × 0.968 = 968Explains the original resultChallenges the chosen inputBoth calculations are arithmetically correct.Only one snapshot satisfies this request.
Figure 2.1. Replaying a calculation establishes what happened. An independent check of the intended inputs establishes why the result was unsuitable.Illustrative single-cash-flow valuation; not an observed trade.

Define the replay contract

To reconstruct a calculation, preserve the inputs and state that actually influenced it. That usually includes an addressable market-data snapshot, instrument and reference-data versions, model and build identifiers, configuration, valuation time, and any relevant prior state.

For an event-driven system, arrival order can matter independently of the timestamp inside an event. Record sequence and processing decisions where they affect state. Make time a supplied dependency where practical. Randomised calculations may also require the generator version, seed or stream state and numerical settings.

Be explicit about the guarantee. Exact reconstruction in the original execution environment is different from numerical agreement within a documented tolerance after a platform change. Intel's numerical-reproducibility documentation explains how floating-point operation order and execution choices can change results.[2] Pinning a model version alone does not settle that question.

A replay runner should control side effects. It must not send a second live quote or repeat a transaction merely because an engineer is investigating the first one. Calculation and external delivery can be separated, or effectful boundaries can be replaced with controlled implementations. Code that writes during calculation is harder to replay safely, but it is not inherently unreplayable.

The record also needs a retention and retrieval design. Preserve enough to support the intended investigations, protect sensitive data, and test whether an older build and its inputs can actually be recovered. “We log everything” is not a substitute for a working reconstruction procedure.

TickTape: one ordered history, the same trading logic

TickTape provides a directly relevant example: a Rust sequencer framework with a price-time-priority order book implemented as a deterministic service. Its service contract requires the same ordered inputs and configuration to produce the same state and outputs. Time is supplied through the execution context as a sequenced timestamp, rather than read from the operating-system clock.[3]

The recovery path applies recorded frames through the service logic and discards replay-generated outputs. That separation matters in trading: reconstructing an order book must not itself resend the historical effects. It is not, by itself, a guarantee of exactly-once delivery to an external venue; the adapter, acknowledgement and reconciliation contracts still matter.[4]

The order-book example uses ordered price levels and FIFO queues. Its invariant checks reject a crossed book and account for accepted quantities as traded, cancelled or still resting, counting a trade on both sides of the ledger. These are checks on trading behaviour, not just on the ability to reproduce bytes.[5]

The boundary remains important. The service contract prohibits ambient time, randomness and I/O; it does not make arbitrary Rust code incapable of using them. Replay equality can expose a violation, but an identically wrong matching rule can pass an equality check. TickTape makes the mechanism concrete without establishing production suitability or model validity. Historical replay still belongs alongside independently checked expected values, constructed edge cases and stress scenarios absent from the recorded period.

FIGURE 2.2TickTape: one history, two execution paths
TickTape: one history, two execution pathsA sequencer orders commands and time into a recorded input history. Live execution and replay use the same service logic. Compare state at the same sequence with the same version, configuration and starting state. Recovery discards generated outputs; external delivery remains a separate contract. CommandsSubmit · cancelSequencerAssign order and sequenced time.ORDERED INPUT HISTORYseq nseq n + 1seq n + 2LIVEREPLAYService logicApply the ordered inputs.The same service logicApply the recorded inputs.State at sequence kState at sequence kCOMPARELive outputs follow the delivery contract.Recovery discards generated outputs.Same service version, configuration and starting state are required.
Figure 2.2. The ordered history makes a trading service reconstructible. Equal state is evidence of replay consistency, not proof that the matching rules or market inputs are correct.Source: TickTape service contract and runtime, commit 0a964bb; see the accompanying implementation citations. Simplified architecture, not a durability or delivery protocol.

Choose failure behaviour before the incident

Missing, delayed and contradictory inputs should have defined meanings. An hour-old observation may be acceptable for one purpose and unusable for another. Freshness depends on the instrument, market conditions and intended action; age alone is not a universal verdict on correctness.

For each important dependency, decide what happens when it is absent, stale or invalid. The response might be to withhold an executable quote, publish a clearly marked indicative value, use an approved fallback, or stop the affected calculation. A degraded mode is useful only when its result remains appropriate for the consumer.

That agreement must extend downstream. A status flag offers little protection if another service discards it or treats an indicative value as executable. Test the consuming workflow, including cached results and delayed messages.

An operator also needs a tested way to stop the relevant activity. Define the scope of the control, who can use it, what happens to work already in flight, and what evidence is required to resume. A switch that stops new calculations but leaves queued output flowing may not provide the intended protection.

The most dangerous architecture is often the one whose failure behaviour was never consciously chosen. It has failure behaviour regardless. It just belongs to whoever wrote the retry loop.

Measure the difficult periods

Average latency can describe ordinary demand or support capacity planning. It is insufficient for understanding deadline misses. A short period of severe delay can disappear inside a session-wide mean.

Google's SRE guidance recommends treating many service indicators as distributions and selecting objectives appropriate to the workload.[6] For pricing, the practical extension is to distinguish interactive quotes from batch valuation, identify the relevant deadline, and inspect both the proportion and the business significance of missed deadlines.

Observe timeouts and rejected work as well as successful responses. If the measurement drops requests that never complete, the latency chart can improve while the service deteriorates. Examine queue age, input freshness and resource use over the same interval so that a fast calculation waiting behind a long queue does not look healthy.

Slow responses can cluster during bursts, dependency failures or resource contention; they do not invariably do so. Test those conditions deliberately. A percentile from a quiet day cannot establish behaviour during a market-data surge.

FIGURE 2.3The average conceals the missed deadline
The average conceals the missed deadlineOf 100 synthetic requests, 95 take 10 milliseconds and five take 200 milliseconds. Bar lengths show request count. Mean latency is 19.5 milliseconds, p99 is 200 milliseconds, and five percent miss a 50 millisecond deadline. REQUEST LATENCYNUMBER OF REQUESTS05010010 ms200 ms95519.5 ms200 ms5%Mean latencyp99 · nearest rankMiss the 50 ms deadline
Figure 2.3. The mean is below the deadline even though five requests miss it. The distribution does not show whether slow requests clustered in time.Synthetic data: 95 requests at 10 ms; 5 at 200 ms. Bar lengths start at zero; percentile uses the nearest-rank convention.

Make durability and delivery explicit

An audit record is useful only if it survives the failure being investigated. Decide which events must be durable before their effects become externally visible, and how consumers identify duplicates or gaps during recovery.

LMAX's own Disruptor guide gives an example in which journalling, replication and business processing receive the same ordered events, with dependencies used to gate processing.[7] The transferable lesson is the ordering requirement. Adopting the same library would not automatically give another system the same recovery properties.

A durable record also does not prove that a downstream consumer received or acted on an output. Preserve the acknowledgements or reconciliation evidence required by the delivery contract. Calculation history and delivery history answer different questions.

Confidence with a defined scope

A trustworthy pricing system makes it possible to establish what happened, assess whether it was acceptable, and identify what remains uncertain. Replay supports investigation. Validation challenges the result. Failure controls limit consequences. Service objectives make timeliness and availability explicit.

These capabilities reinforce one another, but none replaces the others. A system that can explain its failures still needs to meet its operating obligations. The engineering standard is a result that can be used for its intended purpose, with enough evidence to defend that use and enough control to stop when the conditions no longer hold.

Sources

  1. Federal Reserve, OCC and FDIC, “Revised Guidance on Model Risk Management”, 17 April 2026, section V. The SR 26-2 cover letter supersedes SR 11-7 and describes its applicability. Cited for the distinction between reconstruction and validation, not as jurisdiction-specific compliance advice. ↩︎

  2. Intel, “Obtaining Numerically Reproducible Results”, oneMKL Developer Guide, 31 March 2023. This dated edition supports the numerical principle, not a claim about the current API. ↩︎

  3. TickTape, service contract and execution context, source reviewed at commit 0a964bb on 11 September 2026. Service::Config is part of the deterministic inputs; the restrictions on service behaviour are a contract, not a general-purpose sandbox. ↩︎

  4. TickTape, runtime implementation, replay_step and verify_replay. Recovery drains generated outputs; replay verification compares encoded state snapshots. Neither operation validates an external delivery contract. ↩︎

  5. TickTape, order-book implementation, matching semantics and Invariants::check. These are inspected implementation properties, not a claim of production exchange deployment or an independent audit. ↩︎

  6. Chris Jones, John Wilkes and Niall Murphy, “Service Level Objectives”, Site Reliability Engineering, 2016. ↩︎

  7. LMAX Exchange, “LMAX Disruptor User Guide”, “Multicast Events” and “Consumer Dependency Graph,” accessed 10 September 2026. ↩︎

Matt Hartley
Building NetWatch Labs