From received bytes to a historical answer
Two different problems live in this system. A shared-memory bus moves data between processes on one host in roughly a microsecond. A storage route writes bytes to a file, syncs them and records what is committed. They have different costs and different guarantees, and a bus figure is never a recording latency.
The runtime in the audited copy takes normalised trades and full L10 snapshots. A universal live connector does not exist yet; offline FIX, news and the other domains have their own entry paths.
What a receipt means
engine · durable receiptsSix states, deliberately distinguished. Most systems collapse the middle four.
| State | Meaning |
|---|---|
| Accepted in RAM | No durable promise at all |
| Written to the OS | Not necessarily synced |
| Raw synced, receipt not yet delivered | The data may already exist; the sender does not know it yet |
| Prefix committed | Raw and the commit record are stored under the filesystem contract, and the receipt is offered |
| Quality processed | The background check has a verdict — which may be a rejection |
| Compressed | A derived storage form is updated; a separate progress boundary |
A checksum detects damage; it does not restore vanished bytes. fsync and atomic publication are not an audit of every storage device, controller cache or power-loss scenario. “No receipt seen” is also not “definitely not stored” — recovery can find complete records whose reply was lost.
Why group commit exists
engine · group commitSyncing each record separately is expensive. A bounded batch shares the sync and ledger cost across records — and the acknowledgement still waits until the batch is durable.
256 records, 1 MiB payload, a 10 ms flush timer. These are settings, not measured maximum latencies: the timer says when flushing must start, and scheduling and storage still take time after that.
RAM-only batch records can be lost. They were never acknowledged, so they must stay available upstream. The per-record API remains available for callers that want it.
Background work is allowed to fall behind
There is no unbounded event queue in RAM. A bounded, coalescing wake signal points at the real backlog, which lives in the raw journal; the worker reads the committed prefix and does the actual quality, feature and codec work.
processed_records and compressed_through are not the same boundary. A green verdict on old data must never hide a new backlog, so both stay visible.
State is rebuilt from raw into a new generation — not simply resumed from a persistent per-engine checkpoint. Old generations are kept rather than silently dropped.
A compression failure is reported while raw capture continues, and a rebuild stays possible. Tested — but a finite disk still fills, and a slow receipt consumer or clock check can slow the input path.
Not measured is unknown, never zero and never automatically green.
Why one timestamp is not enough
engine · time contractTo know what an algorithm could have known at a historical moment, six questions need separate answers.
| Time | The question it answers |
|---|---|
| Source / broker time | When does the source say the event happened? |
| Local receipt | When did it arrive here? |
| Availability | From when could this input be used here? |
| Feature ready time | When was the derived computation finished? |
| Clock evidence and error interval | How well founded is that local UTC estimate? |
| Monotonic time and epoch | What was the local ordering inside the same clock and boot? |
Illustrative, not a measurement: an event claims source time 12:00:00.090, arrives at
.100 and is available at .103. A backtest at .095 may not use it.
With a 2 ms error margin on availability the estimated upper bound is .105, and strict
visibility before .105 is refused. Valid clock evidence arriving later never retro-fits
trustworthy times onto earlier events, and a missing historical feature-ready time is left missing
rather than invented. Nanosecond units do not imply nanosecond accuracy, and the system can only
assess the evidence an adapter supplies.
Exactness, stated precisely
“Exact” means two different things here, and a codec benchmark can only measure the second one for a selected projection.
Original input bytes are preserved. An invalid JSON line is still recorded as received input; the quality check later says why it is not usable market data.
Specified values, types and metadata read back without precision loss. A price is an integer plus its scale — 10001 at scale 2 is 100.01 — and a ratio can stay a numerator and denominator instead of a rounded result.
Normalised trade and book prices use scaled integers. That is not a project-wide absence of floating point: generic Arrow codecs can preserve float bits, and the B09 quote fixtures deliberately keep original float64 bits.
Read big integers and ns timestamps exactly. A plain JavaScript Number loses precision on both. Arrow exchange checks units, types and metadata rather than silently converting milliseconds to nanoseconds.
Formats, and what each may claim
| Format | Holds | Status |
|---|---|---|
.gqodb.raw | Original received bytes with framing and integrity checks — the base for every recheck and reprocess | Working |
.gqodb.tick | Native tick segments. Shares GQOBHDR1 framing with the book family, but the header selects the adaptive tick codec profile | Working |
.gqodb.ob | Native order-book segments with block index, range read and prefix recovery; header selects the book codec profile | Working |
.gqodb.replay | Offline capture and book replay produced by the CLI | Working |
.gqo.txt | News, documents and typed fundamentals — readable raw or losslessly compressed, revisions kept | Working |
.gqo.model | Lossless archive of an opaque model file with CRC and SHA-256, verified before publication | Working |
| Arrow / Parquet | Explicit exchange contract and benchmark reference, with exact scales and UTC nanoseconds | Working |
Detection uses the magic header, not the extension: renaming a .tick file to .ob changes neither its contents nor its codec, and the facade checks this explicitly.
What the 1 TB limit counts
engine · input accountingA cumulative 1,000,000,000,000 input bytes per data directory over its lifetime — decimal TB, not TiB, not a monthly quota, and not a cap on total disk space. A configuration may lower it.
Reserves its external byte size, whitespace and newlines included. An identical, already-completed re-import is not charged twice.
Counts offered raw bytes — invalid records and resent traffic included. There is no global producer deduplication.
Recovery, recompression, indices and derived output cost no second input charge — but they do cost disk space.
A local ledger, not a tamper-resistant licence server. The same user owns the files. Commercial policy on machines, accounts and migrations is a separate, unsettled question.
Reads that respect the cutoff
engine · causal queryThe standalone query layer separates event time from what was already known, against a fixed capture watermark. It can select the last visible revision, show every revision, or apply tombstones; for features, the availability bounds of declared dependencies count too.
The watermark stops later appended records that happen to share a timestamp from changing a fixed
snapshot: a correction becomes a new revision rather than silently rewriting an older answer. This
holds within the input contract — wrong supplied times, hidden future information in an external file
or an untrustworthy adapter are not made impossible by it. The runtime’s own asof is
narrower: it scans raw history and picks the last qualified event per stream. It is not a SQL engine
and not a large persistent multi-archive index.