Delivered for a racing analytics platform we built and operate, through Platform to Platform and Data to Data.
The challenge. Racing data arrives from inconsistent sources at different times in different shapes, and every new predictive factor a data scientist wants to test has to be built by an engineer first — so the modelling cycle runs at engineering speed, not research speed.
Context
This is the modelling layer underneath the racing platform we build and operate. It generates outcome probabilities in real time for win markets and for multi-leg exotic pools, and it has run in production across multiple jurisdictions for several years.
It exists because a probability engine is only as good as the rate at which you can test new ideas in it, and that rate is usually set by how long it takes an engineer to implement a data scientist’s idea.
The problem
Racing data does not arrive as a dataset. It arrives as three loosely related streams from vendors who disagree with each other.
Pre-race data is form, fields, weights, barriers, track and going, published at varying times before the jump and revised without warning. Post-race data is results, sectionals and margins, arriving after the fact and sometimes corrected days later. Investment data is the odds and money flow, updating continuously until the market closes.
Each vendor has its own identifiers, its own field names, its own idea of what a runner is when it is scratched and re-entered. Before any modelling happens, all of that has to become one structure that a factor can be computed against.
Then the second problem. A predictive factor is a small piece of arithmetic over that structure: a rolling average, a regression residual, a comparison of a runner against the field mean. Data scientists produce these continuously, and in most shops each one becomes an engineering ticket. The modelling cycle then runs at the speed of a sprint board, and most candidate factors never get tested because the cost of testing them is a week of someone else’s time.
Why it is hard
Three specific reasons.
Leakage is easy and silent. A factor computed over both pre-race and post-race data will happily include information that did not exist at bet time, and a model built on it will backtest beautifully and lose money in production. The factor engine has to be able to compute post-race-derived factors for historical context while structurally excluding future values relative to the point being modelled. That constraint has to live in the engine, not in the discipline of whoever wrote the factor.
Factor generation is embarrassingly parallel and expensive. Recomputing a factor library across years of historical racing is a large batch job. Running it on standing infrastructure means paying for peak capacity permanently.
A model that backtests well is not a model you should bet. Whether a model is worth deploying at all is a separate question from how accurately it predicts, and the answer depends on the stake strategy and risk appetite of whoever is betting it, not on the model.
What we built
A 24x7 aggregation service. Core Java, running continuously, acquiring and normalising pre-race, post-race and odds data into one structure in Cassandra. Kafka carries the ingestion so bursts from a vendor do not become back-pressure on the writer.
A factor generation engine with its own expression language. This is the part that changed how the team worked. Rather than exposing a library and asking data scientists to write Java, we built a domain-specific expression language for factor definition: averages, regressions, mean comparisons and composites over the unified racing model. A data scientist writes a factor, the engine compiles and evaluates it, and no engineer is involved. Factors are generated excluding future values relative to the modelled point, placing time-boundary controls in the engine. Data provenance and factor validation remain part of the review.
Parallel processing with dynamic compute allocation. The factor engine distributes work across compute nodes, allocates nodes when a run needs them, and shuts them down when it does not. Cost tracks work rather than tracking peak.
Model development in the tools data scientists already use. Python, R, Stata and H2O, with Python for custom models. We did not build a modelling framework; building one would have meant asking the people doing the modelling to abandon the tools they are fluent in.
A Java backtesting and simulation engine. Historical replay to measure how a model would have behaved against past meetings, plus A/B simulation across competing models for selection. Backtesting is the gate between a model existing and a model being deployed.
A real-time probability engine. Validated models go to a live non-real-money environment where probabilities are generated from models, factors and the latest data as a race approaches. An analytics layer on top maps the best-performing models to customer risk profiles and derives a stake strategy from that mapping, rather than presenting every customer with the same number.
Front end in Next.js, with Elasticsearch indexing model output and results for search-driven analytics and dashboards.
Results
Real-time probabilities for win and exotic pools. Exotics are the harder case, because probabilities for combinations are not independent draws from the win market.
The modelling cycle stopped depending on engineering. The expression language is the whole reason. A factor idea gets tested the day it is had, which changes which ideas get tested at all.
Leakage is structurally prevented rather than caught in review.
Compute cost tracks work. Nodes exist while a factor run needs them.
It ran for years across multiple jurisdictions, which is the result that matters most for a probability system: staying calibrated as fields, tracks and pool behaviour change is a different problem from being accurate once.
We have no accuracy or performance figures cleared for publication. There are none on this page for that reason.
What we would do differently
The expression language should have come first, not second. It was built after the first generation of hand-coded factors, which meant a migration and a period where both paths existed. If the goal is to remove engineering from the modelling loop, the language is the foundation and everything else is downstream of it.
We would also version factors as first-class objects from the start. Models were versioned; factors were treated as definitions. When a factor definition changes, every model built on it silently means something different, and reconstructing which version of a factor produced a given backtest is archaeology.
Where this transfers
The pattern is aggregate inconsistent sources into one model, let domain experts define derived features in a language they can write, prevent leakage in the engine, backtest before deploying, and map the surviving model to the risk appetite of whoever acts on it.
That is the same pipeline as insurance claim probability, demand forecasting against volatile inputs, and market or credit modelling in financial services. The domain changes; the leakage problem and the “who can define a feature” bottleneck do not.