· Protocolzone · updated 21 September 2026
A model that scores well in a notebook and a model that runs a business are different artefacts. The first is a function fitted to a static extract. The second is a service that consumes moving data, produces decisions people act on, degrades quietly as the world shifts under it, and has to be replaceable without breaking anything downstream. Most of the work is in the gap between those two things, and almost none of it is the modelling.
This is a lived account. We have taken three different families of model to production across three domains: a racing probability and ratings engine behind a tier-one wagering platform we build and operate; a wildlife video identification system for a government-sector conservation programme; and a price-forecasting and optimisation model for a food manufacturer. Different maths, different tools, same discipline. The figures below are from our own case-study documents; where we have not measured something, we say so rather than round up.
The first thing that bites: point-in-time data
The failure that is invisible in a notebook and fatal in production is leakage. A feature that quietly encodes information the model would not have had at decision time scores beautifully in backtest and collapses live, because in the notebook the future was sitting in the same dataframe as the past.
On the racing engine, factors are generated with a hard rule: a factor computed for a given race may only use values known before that race. No finishing positions, no post-race adjustments, no market moves that happened after the point we are pretending to stand at. That constraint is enforced in the factor-generation code, not left to the modeller’s memory, because the modeller will forget once and never find out. It is the single most important line of defence between a backtest that means something and a backtest that is lying to you.
The same discipline is what makes the forecasting work trustworthy. A price forecast that trains on data it would not have had on the forecast date is a demonstration rather than a forecast.
A backtest is a claim; a simulation is a rehearsal
Scoring a model on held-out history tells you how it would have done. It does not tell you how the surrounding system behaves when the model’s output starts moving money. For that we built a backtesting and A/B simulation engine in Java that replays historical events through the full decision path, runs a challenger model alongside the incumbent on the same event stream, and compares outcomes on the metrics the business actually cares about rather than on model loss.
The step that catches what backtesting cannot is deploying a candidate into a live, non-real-money simulation before it touches production. It sees real, current events at real timing, produces real decisions, and settles them against reality with no exposure. Timing bugs, feed gaps, and features that are computable in a nightly batch but not in the live path all surface here, where they cost nothing. A model that passes offline backtest and then falls over in live simulation is the normal case, not the exception, and finding out in simulation is the whole point of having one.
Promotion is a gate, never a git push
A model does not reach production because someone is happy with it. It reaches production by clearing a set of gates that are the same for every candidate:
- It beats the current champion on the agreed business metric in backtest, by a margin wide enough to survive noise.
- It runs clean in live non-real-money simulation for a defined window.
- Its features are all confirmed computable at decision time in the production path, not just in the training batch.
- Its inputs and outputs are versioned, so a decision made months ago can be reconstructed from the model version and the data as it stood.
If a candidate cannot clear those, it does not ship, however good the offline numbers look. Writing the gate down is what stops “this one feels better” from becoming a deployment.
Champion and challenger, permanently
Promotion is not the end state. The incumbent is the champion; new candidates run as challengers against the same live event stream, scored on the same metrics, until one earns promotion. The point is that the comparison is always apples to apples: both models saw the same events at the same time, so a difference in outcome is a difference in the model and not an artefact of two runs against two different weeks.
This also gives you a rollback that is already warm. If a promoted model misbehaves, the previous champion is not a redeploy away, it is still running.
Retraining without regression
Models decay because the world moves. The retrained model has to be better on new conditions without being worse on the cases the old one already handled, and that trade-off is easy to lose if the only thing you look at is aggregate accuracy.
The wildlife identification system is the clearest example of doing this well. It ships with an operator-feedback loop: when a human corrects an identification, that correction re-enters the training set, and the next model is validated against both the new corrections and the cases the current model already gets right. The project used more than 20,000 labelled images and reported 96% identification accuracy. The published account does not specify the evaluation split. It also cut manual surveillance workload by roughly 70% by letting operators review flagged frames instead of raw footage. Those are the measured figures from that engagement; we have not published a per-species breakdown because the programme has not cleared one.
The forecasting work carries the same shape in a different domain. Modelling more than 100 product SKUs, the delivered model held above 90% forecast accuracy and drove a 5 to 12% margin improvement in the volatile seasons where a naive forecast hurts most, with the downstream optimisation built on SciPy and Pyomo turning the forecast into an actual pricing and volume decision. A retrain that lifted one SKU family and quietly degraded another would net out flat in the headline number, so the regression check runs per SKU family, not just on the aggregate.
The tools, and what they are and are not for
Across the three domains the modelling itself has been built in Python, R, Stata and H2O, with scikit-learn, XGBoost and Statsmodels for the estimators, and SciPy and Pyomo where the output feeds an optimisation rather than a prediction. TensorFlow and OpenCV sit behind the vision work; FFmpeg handles the video pipeline into it.
The honest point about that list: every one of those is a modelling tool, and none of them is a production system. R and Stata are where a model gets understood; they are not where it gets run against a live feed at the timing production demands. The mistake that costs a quarter is treating the training notebook as the deployment artefact. The model that runs in production is the one wrapped in point-in-time feature generation, versioned inputs, a promotion gate and a champion/challenger harness, and that wrapper is more code than the model.
For the team’s own use of large language models during this work, we run a central, access-controlled gateway that every LLM call goes through, so usage is governed in one place rather than scattered across individual keys and tools. It is internal engineering plumbing rather than a product, and it is mentioned here only because “how do you manage the team’s model access” is a fair question to ask anyone claiming to do this seriously.
What this is, and what it is not
This is a description of how we have shipped models we can point to: a racing engine in production, a vision system delivered to a conservation programme, and a forecasting model delivered to a manufacturer. The figures are the ones those engagements produced and no others. We have not quoted latency, throughput or uptime for any of it, because those are not the numbers these engagements measured, and inventing them to fill a paragraph would undo the only thing this post is trying to establish: that the discipline is real because the deliveries were.
- machine-learning
- mlops
- production
- forecasting
- computer-vision