· Protocolzone
A result gets corrected forty minutes after the race. A scratching lands during the window when two of your providers are mid-update and one is not. An odds tick arrives out of order because a provider retransmitted a batch. Every one of these is normal traffic on a racing feed, and every one of them breaks a pipeline that assumed the data arrives once, in order, and stays put.
This is a note on how we built and run the ingestion layer for a racing data pipeline: a service that pulls pre-race fields, live odds and post-race results from several providers, around the clock, and lands them in a store that downstream models and reporting read from. It is delivered work, in production. The specific domain is racing, but the shape of the problem — many sources that disagree, corrections that arrive late, consumers that must not miss any of it — is the same in any real-time domain, so the design reads across.
The setup
A Core Java service does the ingestion. It talks to each provider in that provider’s own dialect, on that provider’s own schedule, and its single job is to get every message onto a Kafka topic as fast as it arrives. Kafka is the spine. From there the data flows into a Cassandra and ScyllaDB store that feeds model training, live pricing inputs and operational reporting.
We could have wired each provider straight to each consumer. That is the version that looks simpler on the first day and is unmaintainable by the third provider. With N providers and M consumers you are maintaining N×M integrations, every consumer has to understand every provider’s quirks, and there is no one place where a late correction can be applied once and seen by everyone. A log in the middle turns that into N producers and M consumers, each of which only has to understand the topic.
Why a log, specifically
A message queue that drops a message once it is consumed would be the wrong tool here, and it is worth being precise about why.
The value of Kafka in this pipeline is that the topic is a retained, ordered record, not a transient buffer. Three properties fall out of that, and each one maps to a failure we would otherwise have to engineer around.
Replay. A consumer that had a bug, or that we add six months from now, can start from an earlier offset and rebuild its own state from the same history everyone else saw. When a model needs retraining on how the data actually arrived, corrections and out-of-order ticks and all, the tape is still there. A queue that discards on read gives you one shot at every message and no way to reconstruct what happened.
Ordering where it counts. Kafka orders messages within a partition. If we partition by the racing entity, the meeting or the race, then everything about one race arrives at every consumer in the same order it was ingested. That ordering is what lets a consumer reason about “the latest state of this race” without cross-checking against a second source of truth.
Back-pressure that does not lose data. Providers do not send evenly. Barrier draws, jump times and results cluster; the quiet stretches are quiet. When a burst arrives faster than a consumer can process it, the consumer falls behind in offset rather than dropping messages. It catches up when the burst passes. The buffering is the retained log itself, which means a slow consumer degrades into lag instead of data loss.
The problem that shapes everything: corrections arrive after the event
The hard part of racing data is not volume. It is that a fact you already recorded can change after you recorded it. A result is declared, then amended after a protest. A runner shown as a starter is scratched late. A provider retransmits a corrected batch because their upstream fixed something.
If your store holds one mutable row per race and you overwrite it each time, you have thrown away the thing you most need later: what you knew, and when you knew it. So the topic design treats a correction as a new event about an existing entity rather than an edit to a past one. The result declaration and the amended result are both messages. Downstream, the store keeps them as an append with effective timing, so “what was the standing result at 15:42” and “what is the result now” are both answerable questions.
This is also why we do not settle correctness at the consumer. If every consumer had to independently decide how to fold an amendment into its view, they would drift apart, and two reports of the same race would eventually disagree. The fold happens once, at ingestion.
Normalisation happens once, at the edge
Providers disagree on the boring things, and the boring things are what cost you. They use different identifiers for the same horse, meeting or race. They timestamp in different zones and with different notions of “now”. They model a scratching, a gear change or a track condition differently. One sends a full snapshot; the next sends deltas.
The ingestion service resolves all of that before anything reaches the topic. A horse is mapped to one canonical identity regardless of which provider named it. Times are normalised to one clock. Provider-specific status codes are translated into one vocabulary. By the time a message is on Kafka, it is in our shape, and every consumer downstream is spared from ever learning that provider three exists.
Doing this at the edge, once, is the difference between adding a fourth provider being a contained change in one service, and it being a change that ripples into every model and every report that reads the data.
The payoff downstream: leakage-safe features
There is a specific reason all of this timing discipline matters for the models that read the pipeline. A racing model that learns to predict an outcome must be trained only on information that existed before that outcome was known. If a corrected result, declared after the race, leaks into the features used to predict that same race, the model looks brilliant in backtest and is worthless in production. It has learned to read the answer.
Because the pipeline records what was known and when, feature computation can be pinned to a point in time and can exclude anything stamped after it. The correction is still in the store, because we need it for settlement and reporting, but it is fenced off from any feature that represents a pre-race state. The event-time discipline at ingestion is what makes that fence possible; you cannot enforce it downstream if the upstream already overwrote history.
What this is, and what it is not
This is a working ingestion pipeline we operate, in racing. The claims here are about that: the log-shaped architecture, the correction-safe topic design, normalisation at the edge, and the leakage guard it enables.
It is not a general prescription that Kafka belongs in every pipeline. If you have one provider, few consumers, and no late corrections, a log is machinery you do not need yet. The design earns its place at the point where sources multiply and disagree, and where a fact can change after you first recorded it. Racing hits both early. Trading feeds, sensor telemetry and any multi-source real-time domain hit them too, which is why the pattern moves across without much translation.
- kafka
- data-engineering
- racing
- event-streaming
- cassandra