This is a financial & insurance problem we approach through our Data to Data service line.
The problem
You buy data from more than one vendor, because coverage and redundancy demand it. Each vendor has its own symbology, its own timestamps, its own publishing schedule and its own idea of what a correction is. One re-keys instruments without notice. One restates yesterday’s values inside tomorrow’s file. One publishes the same event twice with different precision.
Downstream, every consumer — each model, each desk tool, each report — does its own partial reconciliation of that disagreement, differently. The disagreements between your vendors become disagreements between your own systems, and the question “what did we know at 14:32 last Tuesday” has as many answers as you have consumers.
This is the unglamorous majority of trading infrastructure. The strategy work gets the attention; the pipeline that makes disagreeing feeds agree, stay point-in-time correct and replay on demand is what the strategy work silently depends on.
Why it’s hard
Identifier mapping is temporal, not a lookup table. Symbols get reused, instruments move venues, vendors re-key their universes. A static mapping decays without announcing it — the failure appears weeks later as a model quietly computing over the wrong instrument’s history. The mapping has to be versioned data, so that resolution is reproducible for any past date.
Corrections destroy history if you let them. The natural implementation — apply the vendor’s correction in place — means every backtest run after the correction is computed against information nobody had at the time. The corruption is invisible: nothing errors, results just stop being honest. Corrections have to be appended events over an immutable record, which is a storage-model decision you cannot easily reverse out of later.
Time is plural. Every event has an event time, a vendor publish time and your arrival time, and they disagree — by seconds normally, by much more when a vendor recovers from an outage and replays a backlog. Joining streams on the wrong time axis produces subtly wrong context for every downstream computation, and it is the kind of wrong that backtests do not surface.
Bursts are the normal case. Feeds are quiet and then simultaneous. A pipeline sized for average throughput back-pressures at exactly the moments that matter most, and a writer that stalls under a burst drops or delays the data the whole desk is waiting on.
It never gets to stop. A feed fails at 3am, resumes with a gap, and replays out of order. Ingestion is an operations commitment as much as an engineering artefact — the pipeline needs monitoring, alerting and someone who answers, permanently.
How we approach it
Continuous acquisition into one canonical structure. Core Java services run 24x7, acquiring each vendor’s feed and normalising it — identifiers, field names, units, correction semantics — into a single canonical model in Cassandra. Consumers integrate once, against one structure, instead of each carrying per-vendor logic.
Kafka between acquisition and the writer. Ingestion is carried on Kafka so a burst from one vendor is absorbed by the log rather than becoming back-pressure on the write path. Replaying a stretch of ingestion is then an offset decision, not a recovery project.
Corrections as appended events. The stored record is immutable; vendor corrections and restatements append to it. The structure can answer both “what is correct” and “what was known at time T”, which is the property everything in backtesting rests on.
Versioned identity mapping. Vendor identifiers resolve through a mapping that is itself historised, so instrument identity is reproducible for any date, including across re-keys.
Monitoring as a first-class deliverable. Prometheus and Grafana over feed liveness, latency, gap detection and per-vendor disagreement rates — because a normalisation pipeline that fails silently is worse than no pipeline, and our 24×7 Operations Desk runs this class of system as a standing service.
What we do not do. We are not a market data vendor and we do not resell data — licensing sits between you and your vendors, and redistribution terms in those agreements constrain what a shared structure may serve to whom, so they belong in the design conversation from the first week. We would also not normalise away vendor disagreement invisibly: where two vendors disagree, the disagreement is recorded and a stated precedence rule resolves it, so the resolution is auditable instead of accidental.
What it takes
Your vendor agreements, early. Entitlements and redistribution clauses decide which consumers may read which fields from a unified structure. This shapes the design and is cheaper to know before the schema exists.
A canonical model decision. Someone has to own what an instrument, an event and a correction are in your world. This is a fortnight of arguments that saves a year of reconciliation, and it cannot be delegated to the vendor with the loudest documentation.
Per-vendor correction semantics, written down. Each vendor’s rules for restatement, replay and gap recovery have to be captured explicitly, because the pipeline encodes them. The exercise usually reveals that nobody currently knows what one of your vendors actually does.
An operations arrangement. A 24x7 pipeline needs 24x7 coverage — either your team’s, or ours.
Where this has been done
Delivered — against racing data, which is the same problem with different nouns. The racing analytics platform we built and operate ingests three stream families from vendors who disagree with each other: pre-race data published at varying times and revised without warning, post-race results corrected days later, and continuously updating odds and investment data. A 24x7 Core Java aggregation service normalises all of it into one structure in Cassandra, with Kafka carrying ingestion, and it has run in production for several years. The racing probability modelling case study covers the pipeline and the modelling stack that sits on it.
Nearer to this page’s domain: AnkEDGE, our own options trading platform built for options traders, is engineered on the same disciplines. We have not delivered this pipeline for a market data estate at a financial institution, and the claim here is the pattern — disagreeing time-stamped feeds, made to agree, kept point-in-time correct, replayable — demonstrated in production, not a financial-markets reference.