This is a financial & insurance problem we approach through our Data to Data service line.
The problem
Your research team produces candidate strategies faster than your production results can validate them. A strategy backtests strongly, goes live, and underperforms — and now you have a diagnosis problem stacked on a performance problem. Did the market regime change? Did execution costs eat the edge? Or was the backtest lying from the start, and the strategy never had an edge to lose?
Most desks cannot answer that question, because the backtest and the live system compute their inputs differently, and nobody can reconstruct exactly what the model would have known at any historical decision point.
There is now a second audience for the answer. Under MiFID II Article 17 and RTS 6, an EU firm running algorithmic strategies must maintain a documented development and testing methodology (Article 5), test in non-live environments segregated from production (Article 7), and produce an annual self-assessment (Article 9) — which ESMA’s February 2026 supervisory briefing now expects to cover the impact of AI. In Australia, ASIC RG 241 requires review and certification of automated order processing systems before use, and CP 386 (consultation closed October 2025) proposes principles-based rules for algorithm development, testing and monitoring however orders are generated. Backtesting rigour has become something a regulator asks to see, not only something P&L punishes you for skipping.
Why it’s hard
Leakage is easy and silent. A factor computed over the full history will happily include information that did not exist at decision time — a settlement price in a feature evaluated intraday, a restated fundamental in a feature evaluated before the restatement. The failure mode is specific: the backtest improves, live performance does not, and nothing in the backtest output tells you which factors are leaking. Code review does not catch it reliably because the leak is in the data relationship, not the code.
Your historical data is not what you would have seen. Vendors correct prints, restate values and revise history without ceremony. A backtest run against today’s copy of history is a backtest against information you did not have. Point-in-time correctness — what was known when, not what turned out to be true — has to be a property of the data layer, or every result inherits the corruption.
Survivorship bias arrives by default. Backtesting today’s instrument universe against past markets silently excludes everything that delisted, defaulted or was withdrawn. The surviving universe outperforms by construction.
Enough experiments guarantee a false positive. Test a hundred variants and several will clear any fixed performance bar by chance. This is the selection bias that Bailey and López de Prado’s deflated Sharpe ratio (Journal of Portfolio Management, 2014) exists to correct, and it means the number of trials is itself an input the validation has to know about.
A model that predicts well is still not automatically a model you should trade. Deployment depends on the stake strategy and the risk appetite of whoever acts on the output. That is a property of the deployer, and no backtest metric answers it.
Recomputing history is expensive. Replaying a factor library across years of data is a large parallel batch job. On standing infrastructure you pay for peak capacity permanently to serve occasional runs.
How we approach it
The leakage guard lives in the engine. Factors are generated with future values structurally excluded relative to the point being modelled. The engine restricts factor evaluation to information available at the decision point. This reduces dependence on each factor author remembering the boundary. Input timestamps, derived features and historical corrections still need validation to establish that a particular backtest is free of leakage.
Walk-forward replay against point-in-time data. Models are evaluated by replaying history as it stood at each decision point and rolling forward, so out-of-sample discipline is the default shape of a test rather than an option. Corrections are preserved as corrections, not overwrites, so a result computed last year is reproducible this year.
A/B simulation across competing models. Selection between candidate models happens in the same replay machinery, against the same data, so the comparison is between models rather than between test harnesses.
A live simulation stage before production. Surviving models are deployed to a live non-real-money environment, generating outputs from current data in real time. This is the gate that catches what historical replay structurally cannot: feed timing, operational behaviour, and the difference between a model’s paper decisions and the decisions it makes against a moving market.
Model output mapped to risk appetite. The analytics layer maps validated models to the risk profile of whoever acts on them and derives sizing from that mapping, keeping “is the model good” and “should we act on it” as the separate questions they are.
Compute allocated per run. Factor recomputation distributes across nodes allocated for the run and released after it, so cost tracks work.
What we do not do. We do not sell strategies, signals or alpha — the models are yours, and we build the machinery that tests them honestly. We are not a licensed financial adviser and nothing in the engagement is investment advice. And the machinery does not discharge your regulatory obligations: RTS 6 keeps testing responsibility with the firm even when systems are vendor-supplied. What it produces is the evidence trail your self-assessment is made of.
What it takes
Point-in-time historical data, with corrections preserved. If your history is a single mutable copy, building the bitemporal layer is the first project, not an optional one. Expect this to be where most of the calendar goes.
A factor dictionary with declared information timing. Every input needs an owner and an answer to “when does this value become knowable”. That inventory usually reveals a handful of factors nobody can source cleanly, which is better discovered now than in production.
Trial accounting. Validation that corrects for selection bias needs to know how many experiments were run, which means recording the failures, not just the survivors.
A written deployment gate. Who promotes a model from simulation to production, against what criteria, recorded where. For regulated firms this is also the artefact your RTS 6 Article 9 self-assessment or ASIC certification review will ask for, so it is worth writing once, properly.
Where this has been done
Delivered — in racing markets, not financial ones. We built and operated the backtesting and simulation layer of a racing analytics platform: a Java replay and A/B simulation engine, a factor engine that excludes future values structurally, and a live non-real-money simulation stage ahead of production, running continuously for several years across multiple jurisdictions. The racing probability modelling case study covers it.
The transfer to financial markets is mathematics, not marketing: point-in-time data discipline, survivorship (a scratched runner and a delisted stock are the same bias), leakage, regime change and calibrated probabilities under a moving market are the same problems with different nouns. We have not shipped this stack for an equities or futures desk, and this page does not pretend otherwise.