This is a sports & gaming problem we approach through our Data to Data service line.
The problem
You have three or four data providers. One sends pre-race form and fields, one sends results, one sends prices and investment. They disagree about runner identifiers, they disagree about what a scratching is, and one of them sends corrections hours after the race.
Somewhere downstream, a trading team or a modelling team is meant to produce a number they will price a market off, or bet into a pool with. What they actually have is a set of spreadsheets, a nightly job that half works, and a model whose backtest nobody trusts because nobody can reproduce the dataset it was trained on.
The gap is not modelling talent. It is that there is no dataset with a stable shape, and no way to get a new idea from a modeller’s head into something testable without an engineering release.
Why it’s hard
Reconciliation is the actual work, and it is unglamorous. Two providers name the same runner differently, a late gear change arrives after fields are published, a result is amended after protest. Every one of those has to be absorbed without breaking the identifier a model was trained against.
Look-ahead bias is the failure mode that hides longest. Post-race data is the richest source you have for building factors, and the moment any of it leaks into a factor for a race that has not run, the backtest becomes fiction. It does not error. It gets better. This is why point-in-time correctness has to be a property of the factor engine rather than a discipline you ask analysts to maintain.
Exotic pools need a joint distribution, not a ranking. A win probability per runner is not enough to price a trifecta. A trifecta in a fourteen-horse field has 2,184 ordered combinations, and the probability of each depends on the others through whatever dependence assumption you choose. Rating systems that stop at the win market stop there because this is where the arithmetic gets expensive.
There is a deadline. Prices, scratchings and pool investment move until the jump. A probability set computed on the morning’s data is stale by the time it matters, so the pipeline has to regenerate factors and re-run models continuously rather than in a nightly batch.
Compute cost is concentrated. Factor generation across a full historical archive is a large parallel job that you need for a few hours and then do not need at all. Provisioning for the peak and leaving it running is how a modelling programme becomes too expensive to iterate on.
How we approach it
A 24×7 aggregation service, not a nightly import. We run ingest as a continuously operating service in Core Java, landing pre-race, post-race and odds data as it arrives and normalising it onto one internal model with stable identifiers. Kafka carries the ingest stream. The normalised history lands in a wide-column store — Cassandra on the original build, ScyllaDB where we need the same model with tighter latency — because the access pattern is “everything about this meeting, this runner, this period” rather than relational joins.
A factor engine with its own expression language. Factors are defined in a domain-specific expression language rather than in application code. That is the design decision that matters most on this kind of programme: it moves factor authorship from the engineering backlog to the modelling team, and it lets the engine enforce point-in-time semantics centrally instead of trusting each author to remember them. Averages, regressions, mean comparisons and compositions of those over configurable windows all express in the language.
Compute that scales up and then goes away. Factor generation runs in parallel across compute nodes that are allocated for the job and shut down when it finishes. Historical regeneration becomes a cost you pay per experiment rather than a standing bill.
Modelling in whatever the modellers use. Python, R, Stata and H2O all read the same factor set. We do not impose a modelling stack; we impose a dataset contract.
A simulation engine before anything goes live. A Java-based backtesting and simulation engine evaluates a candidate model against history, and supports A/B comparison across models rather than a single pass/fail. Models that survive that go into a live, non-real-money environment generating probabilities against real-time data, so their behaviour under live conditions is observed before money is at risk.
Serving and analytics. Probabilities and model diagnostics are indexed in Elasticsearch and delivered through a Next.js front end, so a modeller inspecting why yesterday’s model behaved oddly is running a query rather than filing a request.
What we would not do. We would not put a model into production off a backtest alone; the live simulation stage exists because backtests flatter. We would not build a monolithic “prediction service” that owns both the data and the models — the value is in separating a stable factor substrate from models that change weekly.
What it takes
Historical depth, and permission to use it. Factors are only as good as the archive behind them. If your providers’ historical files are incomplete or your licence does not cover model training, that is the first conversation, not a detail.
Someone who owns the models. This platform makes a modelling team fast. It does not replace one. Engagements where the client has no modelling capability work best when we build the first generation of models and hand over both them and the expression language.
A cloud account shaped for burst. The economics depend on being able to bring up and tear down compute nodes freely.
Acceptance that the first useful output is a diagnostic, not a price. The early value is discovering which of your existing assumptions the data does not support.
Where this has been done
This is a shipped reference, not a capability claim. We built and operated a real-time probability platform of exactly this shape as part of Ashva, our own racing wagering platform, and it ran in production across multiple jurisdictions for several years, generating probabilities for both win and multi-leg exotic pools. The delivery sits under commercial confidentiality, so this page describes the architecture rather than the operator.
The transferable part is the architecture, not the racing. The same substrate — continuous ingest, point-in-time factor generation, a modelling language, and a simulation stage between backtest and production — is what we build for forecasting and anomaly work in other domains.