This is a financial & insurance problem we approach through our Data to Data service line.
The problem
The model was the part that went well. It backtested convincingly, the notebook is persuasive, and a version of it is in production making decisions that move money or commit the organisation.
Then the questions start. Accuracy this quarter is below the backtest and nobody can say when the decay started. The team wants to retrain, but the last retrain made one segment worse and there was no way to know before it shipped. An auditor, or a regulator, or your own risk function asks why the model scored a specific case the way it did in March, and the honest answer is that March’s model no longer exists anywhere it can be run.
None of these are modelling problems. They are lifecycle problems: how training data is constructed, how a candidate is proven better than the incumbent, how retraining happens without regression, and how a historical decision stays reproducible. In model-risk terms — the SR 11-7 vocabulary most financial institutions borrow — it is the difference between having a model and having model risk management. The model is the easy part. The lifecycle is the product.
Why it’s hard
Leakage is silent and flattering. A feature computed over data that includes anything from after the decision point produces a backtest that is excellent and a production model that is not, and nothing in the accuracy numbers tells you which you have. Financial data makes this worse than most domains: results, corrections and settlements are backfilled onto historical records, so a naive join against “the data we have now” is a join against the future. Discipline does not fix this, because the failure is invisible when it happens.
Data drift and concept drift decay a model differently. Drift in the input population — new customer cohorts, a changed market regime — shows up in population stability metrics before it shows up in outcomes. Concept drift, where the same inputs start meaning something different, shows up only in outcome performance, weeks later. Monitoring built for one misses the other, and a fixed retraining cadence is mistimed for both.
Retraining is a regression risk with no free test. A retrained model is a different model. Without the ability to replay history through both the incumbent and the candidate on identical inputs, “better overall” can conceal “worse on the segment that matters”, and the first place you find out is production.
Backtest accuracy is not deployment fitness. Whether a model should act at all depends on the decision policy wrapped around it — thresholds, sizing, risk appetite — not on the model alone. A model that predicts well can still lose money or make bad calls under the wrong policy, and a promotion gate that only measures accuracy cannot see that.
Reproducibility decays by default. Model files get overwritten, feature definitions change under old models silently, and overrides live in chat history. Unless versions, features, inputs and overrides are recorded as first-class objects, the ability to reproduce a decision erodes without any single event you could have caught. ASIC’s REP 798 review of AI use among licensees flagged exactly this gap: governance lagging adoption, with a credit-risk model nobody could explain as the cautionary example.
How we approach it
Point-in-time data construction, enforced by the engine. Training and backtesting data is assembled so that every feature is computed only from values that existed at the moment being modelled. In the racing probability engine we built and operate, the factor generation engine applies this structurally: data scientists define new factors in a purpose-built expression language, and the engine excludes future values relative to the modelled point regardless of how the factor is written. Leakage prevention belongs in the machinery, not the review checklist.
Feature definitions as versioned objects. When a feature definition changes, every model built on it quietly changes meaning. We version factors and features so a backtest is reconstructable against the exact definitions it ran on — a lesson we learned by not doing it first and having to do the archaeology.
A promotion path with gates, in order. Backtest against historical replay; A/B simulation across candidate models on the same inputs; then a live simulation environment where the candidate consumes production data in real time without acting on it. The racing engine runs precisely this sequence — backtest, competitive simulation, live non-real-money environment — and a model reaches production only by winning through it. Champion–challenger is the standing state, not a migration event.
Retraining driven by triggers and fed by operators. Drift monitoring on inputs and outcomes sets the trigger. The training data for the next pass comes, wherever the workflow allows it, from the humans already reviewing the output: in the wildlife identification system we delivered, rangers validating or correcting each identification generated the labelled data for the next retraining cycle, on a base of 20,000+ labelled images. The model improves from the same activity that consumes it, which is what keeps a retraining loop funded after the project team moves on.
Incremental extension over rebuilds. The identification model was built to take new individuals and new imagery as incremental retraining rather than a rebuild, because the class set grows in production. The same principle applies to a fraud typology or a customer segment that did not exist at training time.
An audit trail as an output, not an aspiration. Every score carries its model version. Every override is recorded with who and why. Reproducing a historical decision is a replay, and the explanation a reviewer needs exists because the system wrote it down at decision time.
What we do not do. We do not sell a monitoring dashboard and call it lifecycle management — a drift chart with no gated path from alert to retrained, validated, promoted model is observability without consequence. And we do not automate the promotion decision itself: gates produce evidence, a person accountable for the model acts on it.
What it takes
Historical data with timestamps you trust. Point-in-time reconstruction is only as good as knowing when each value became known. If your warehouse overwrites in place, the first work package is building the bitemporal record — and that work is reusable well beyond the models.
A replay harness before a new model. The ability to run history through incumbent and candidate on identical inputs is the prerequisite for every gate. It is typically weeks of engineering and it is the highest-value item in the programme.
An owner for each model. Promotion gates produce evidence for a decision someone accountable has to make. Tiering your model inventory — which models warrant independent validation, which warrant monitoring only — is a risk-function conversation we can inform but not have for you.
Reviewers whose corrections can be captured. If humans already check the output, the retraining loop is nearly free; the work is capturing their corrections as structured labels. If nobody reviews the output, that is a finding about the model’s risk tier, not a tooling gap.
Patience with the boring half. The registry, the versioning, the replay harness and the drift monitors are most of the engineering and none of the demo. Budgeting them as first-class deliverables is the difference between a model programme and a notebook graveyard.
Where this has been done
These are shipped, anonymised references, and the pattern held across three domains.
In racing, we built and operate the probability modelling stack behind a wagering platform: leakage-guarded factor generation through a purpose-built expression language, backtesting and A/B simulation across competing models, and a live simulation environment gating promotion to production. It ran for years across multiple jurisdictions — the durability claim, staying calibrated as conditions change, is the one that matters for a lifecycle.
In vision, the wildlife identification system we delivered in 2018 reached 96% individual identification accuracy on a base of 20,000+ labelled images, with an operator feedback loop in which corrections became the next round of training data.
In forecasting, the price forecasting engine we built for a manufacturer was fed daily arriving market data, with forecast confidence derived from historical volatility.
The linked case studies provide the engagement-specific results. The forecasting case reports above 90% forecast accuracy and a 5–12% margin improvement across more than 100 SKUs. These historical ML results do not measure generative AI or predict the performance of a new implementation.