Protocolzone — Insights

AI-generated proposals need an approval path

Mon, 21 Sep 2026 00:00:00 GMT

A language model can turn source material into a plausible proposal. The harder engineering question is what permits that proposal to enter the business system.

In an AI integration we have implemented, generation, review and publication are separate steps. This article describes the implementation and its test coverage in source; it is not a claim of a customer production rollout or a measured business result.

Generation creates a review item

The generator stores a draft with a pending-review status. Creating the draft does not publish it to the platform. That separation gives the application a place to validate the response and a person a place to make a decision.

The stored evidence includes the source item, the prompts used and the raw model response. Keeping those inputs matters: reconstructing a prompt later from the current code does not establish what the model saw when it produced the answer.

Approval is a state transition

The review operation records an approver identifier and changes only a draft that is still pending. If two reviewers act on the same item, the guarded update lets the application detect that the other decision has already happened.

Rejection is retained rather than deleting the item. When the same source is encountered again, the existing decision can be recognised instead of placing the rejected proposal back into the queue.

This workflow also needs an authenticated identity boundary before operational use. A supplied actor label is not proof of identity. The reviewed module keeps its administrative access closed by default while that integration is completed.

Publishing has its own failure handling

A separate publisher selects approved items. It waits for the message send to succeed rather than assuming that handing a message to an asynchronous client means the platform received it.

A failed send leaves the approval available for another attempt. Stable proposal identifiers support downstream duplicate handling. This is a retry design, not a blanket promise of exactly-once delivery across a database and message broker.

Test the boundary, not just the prompt

The source includes tests for pending drafts being withheld, approval requiring an actor, conflicting decisions, rejected drafts and failed message sends. These exercise the workflow around the model; they do not establish model accuracy or replace deployment validation.

The lesson is practical: specify what AI may propose, who may approve it, what evidence is retained and how interrupted work resumes. Those decisions are part of the implementation, alongside the model and its evaluation.

Discuss AI inside your existing workflow.

Multi-tenant wallets: a timeout is not a failed debit

Mon, 21 Sep 2026 00:00:00 GMT

When a platform calls an operator's wallet, a timeout leaves an important question unanswered: did the debit fail, or did it succeed and lose its reply? Retrying both situations in the same way can create a second financial action.

Our wallet integration code treats that uncertainty as a separate state. This is an engineering account of the implemented design, not a published customer rollout or a claim about transaction volumes.

Record the instruction before dispatch

The gateway stores the instruction before making the external call. That gives recovery logic a record to inspect if the process stops while a request is in flight. Recording only the response would leave the most difficult case with the least evidence.

Guarded state updates distinguish work awaiting dispatch from work already sent. The implementation checks whether it acquired the right to act on that record before making the call.

Keep an uncertain debit distinct

For this wallet contract, an uncertain debit is not sent again automatically. It enters an unknown state and is handed to rollback handling. Credits and rollbacks have their own retry behaviour, with stable identifiers carried through the attempts.

The distinction depends on the remote wallet's contract. A rollback needs a defined meaning even when the original response never arrived, and duplicate requests must have a defined response. These are integration requirements to agree with the wallet provider, not assumptions a retry library can settle.

Limit work by tenant

The implementation also bounds concurrent wallet calls separately for each tenant. A slow dependency consumes that tenant's allowance instead of occupying all of one shared allowance.

This reduces one route by which a slow operator can affect others. It does not remove the need to size and monitor shared connection pools, memory and message processing capacity.

What this means for platform development

Multi-tenancy reaches beyond adding a tenant identifier to a table. It affects external calls, recovery state, concurrent work and operational responsibility. Those boundaries belong in the design of the platform and its integrations.

For a new integration, agree the outcomes before writing the happy path: success, explicit rejection, uncertain response, duplicate request and recovery after a restart. Then test the actual provider against that contract.

Discuss your platform integrations.

What it takes to run a model in production, not just train one

Wed, 02 Sep 2026 00:00:00 GMT

A model that scores well in a notebook and a model that runs a business are different artefacts. The first is a function fitted to a static extract. The second is a service that consumes moving data, produces decisions people act on, degrades quietly as the world shifts under it, and has to be replaceable without breaking anything downstream. Most of the work is in the gap between those two things, and almost none of it is the modelling.

This is a lived account. We have taken three different families of model to production across three domains: a racing probability and ratings engine behind a tier-one wagering platform we build and operate; a wildlife video identification system for a government-sector conservation programme; and a price-forecasting and optimisation model for a food manufacturer. Different maths, different tools, same discipline. The figures below are from our own case-study documents; where we have not measured something, we say so rather than round up.

The first thing that bites: point-in-time data

The failure that is invisible in a notebook and fatal in production is leakage. A feature that quietly encodes information the model would not have had at decision time scores beautifully in backtest and collapses live, because in the notebook the future was sitting in the same dataframe as the past.

On the racing engine, factors are generated with a hard rule: a factor computed for a given race may only use values known before that race. No finishing positions, no post-race adjustments, no market moves that happened after the point we are pretending to stand at. That constraint is enforced in the factor-generation code, not left to the modeller's memory, because the modeller will forget once and never find out. It is the single most important line of defence between a backtest that means something and a backtest that is lying to you.

The same discipline is what makes the forecasting work trustworthy. A price forecast that trains on data it would not have had on the forecast date is a demonstration rather than a forecast.

A backtest is a claim; a simulation is a rehearsal

Scoring a model on held-out history tells you how it would have done. It does not tell you how the surrounding system behaves when the model's output starts moving money. For that we built a backtesting and A/B simulation engine in Java that replays historical events through the full decision path, runs a challenger model alongside the incumbent on the same event stream, and compares outcomes on the metrics the business actually cares about rather than on model loss.

The step that catches what backtesting cannot is deploying a candidate into a live, non-real-money simulation before it touches production. It sees real, current events at real timing, produces real decisions, and settles them against reality with no exposure. Timing bugs, feed gaps, and features that are computable in a nightly batch but not in the live path all surface here, where they cost nothing. A model that passes offline backtest and then falls over in live simulation is the normal case, not the exception, and finding out in simulation is the whole point of having one.

Promotion is a gate, never a git push

A model does not reach production because someone is happy with it. It reaches production by clearing a set of gates that are the same for every candidate:

It beats the current champion on the agreed business metric in backtest, by a margin wide enough to survive noise.
It runs clean in live non-real-money simulation for a defined window.
Its features are all confirmed computable at decision time in the production path, not just in the training batch.
Its inputs and outputs are versioned, so a decision made months ago can be reconstructed from the model version and the data as it stood.

If a candidate cannot clear those, it does not ship, however good the offline numbers look. Writing the gate down is what stops "this one feels better" from becoming a deployment.

Champion and challenger, permanently

Promotion is not the end state. The incumbent is the champion; new candidates run as challengers against the same live event stream, scored on the same metrics, until one earns promotion. The point is that the comparison is always apples to apples: both models saw the same events at the same time, so a difference in outcome is a difference in the model and not an artefact of two runs against two different weeks.

This also gives you a rollback that is already warm. If a promoted model misbehaves, the previous champion is not a redeploy away, it is still running.

Retraining without regression

Models decay because the world moves. The retrained model has to be better on new conditions without being worse on the cases the old one already handled, and that trade-off is easy to lose if the only thing you look at is aggregate accuracy.

The wildlife identification system is the clearest example of doing this well. It ships with an operator-feedback loop: when a human corrects an identification, that correction re-enters the training set, and the next model is validated against both the new corrections and the cases the current model already gets right. The project used more than 20,000 labelled images and reported 96% identification accuracy. The published account does not specify the evaluation split. It also cut manual surveillance workload by roughly 70% by letting operators review flagged frames instead of raw footage. Those are the measured figures from that engagement; we have not published a per-species breakdown because the programme has not cleared one.

The forecasting work carries the same shape in a different domain. Modelling more than 100 product SKUs, the delivered model held above 90% forecast accuracy and drove a 5 to 12% margin improvement in the volatile seasons where a naive forecast hurts most, with the downstream optimisation built on SciPy and Pyomo turning the forecast into an actual pricing and volume decision. A retrain that lifted one SKU family and quietly degraded another would net out flat in the headline number, so the regression check runs per SKU family, not just on the aggregate.

The tools, and what they are and are not for

Across the three domains the modelling itself has been built in Python, R, Stata and H2O, with scikit-learn, XGBoost and Statsmodels for the estimators, and SciPy and Pyomo where the output feeds an optimisation rather than a prediction. TensorFlow and OpenCV sit behind the vision work; FFmpeg handles the video pipeline into it.

The honest point about that list: every one of those is a modelling tool, and none of them is a production system. R and Stata are where a model gets understood; they are not where it gets run against a live feed at the timing production demands. The mistake that costs a quarter is treating the training notebook as the deployment artefact. The model that runs in production is the one wrapped in point-in-time feature generation, versioned inputs, a promotion gate and a champion/challenger harness, and that wrapper is more code than the model.

For the team's own use of large language models during this work, we run a central, access-controlled gateway that every LLM call goes through, so usage is governed in one place rather than scattered across individual keys and tools. It is internal engineering plumbing rather than a product, and it is mentioned here only because "how do you manage the team's model access" is a fair question to ask anyone claiming to do this seriously.

What this is, and what it is not

This is a description of how we have shipped models we can point to: a racing engine in production, a vision system delivered to a conservation programme, and a forecasting model delivered to a manufacturer. The figures are the ones those engagements produced and no others. We have not quoted latency, throughput or uptime for any of it, because those are not the numbers these engagements measured, and inventing them to fill a paragraph would undo the only thing this post is trying to establish: that the discipline is real because the deliveries were.

Blockchain and the tote: what a provably-fair pool would take

Wed, 02 Sep 2026 00:00:00 GMT

A punter backs a runner into a win pool. The gates open, the race runs, and a minute or two after the judge's call a dividend appears: $4.30 the win. They are paid on it. What they never saw is the pool that number came out of — how much total money went on their runner, how much on the field, what deduction the host took before dividing the rest. The dividend is an assertion by the tote host, and the punter's only options are to trust it or to lodge a query they cannot independently check.

That gap is worth sitting with, because pari-mutuel is one of the few wagering products where the maths is genuinely public-spirited by design. Nobody prices against you. The host does not carry a book. Every dollar staked on a pool is divided among the winners of that pool after a fixed deduction, so the "odds" are just an accounting identity over money that punters themselves put in. The computation is well understood. What punters cannot see is the pool it runs on.

This post is a design position, not a product announcement. We build and operate a multi-tenant tote and fixed-odds platform, and mechanically that platform routes bets to external tote hosts — the host computes the dividend, we carry commission and no pool-outcome risk. We do not commingle pools, apply breakage, or declare dividends. That is exactly why on-chain pool transparency is interesting to us rather than threatening: the part a chain would make verifiable is the part we already do not own.

What pari-mutuel actually asserts

Strip a win pool to its arithmetic and there are four claims the host is making, in order:

Total staked on the pool: the gross, across every selling point that fed into it.
The deduction: takeout, plus any statutory levy, removed before division.
The winning-unit count: how many dollars are on the runner(s) that get paid.
The dividend: net pool divided by winning units, then rounded down to the breakage increment.

Each of those is a number a punter has to take on faith. Commingling makes it harder still: pools from several jurisdictions merge into one host pool, so the gross you are dividing into includes money you never saw sold and closed on a clock set somewhere else. Amendments after a protest can reopen all four. The honest summary is that the fairest-by-design product in wagering ships with the weakest audit trail for the person whose money is in it.

The real blockchain work we are doing, and what it is not

We have production-grade blockchain engineering underway, but it is not a tote. It is our own NFT-and-wallet ecosystem, and the work that has taught us anything worth writing down is the unglamorous half: taking a wallet from "it signs on a testnet" to something that can hold value in production. Key custody and the signing boundary. State that is aware of confirmation depth, so a balance the application shows is qualified by how many blocks deep the transaction that created it now sits. Handling the case where a transaction you treated as settled is no longer on the canonical chain.

That last problem is the bridge to the tote question. A chain does not give you a single moment of truth; it gives you a probability of truth that hardens as blocks accrue on top. Everything below follows from taking that seriously.

We have not built a blockchain tote. What follows is where the pieces line up, and where they do not.

Where a chain lines up with the pool

The four assertions above map almost cleanly onto operations a chain is good at.

Stake as an append-only record. Every bet into a pool is a write. If those writes land on a shared ledger, the gross is not a number the host reports at close; it is a sum anyone can recompute from entries they can see. A punter checking that their own stake is in the total becomes a lookup rather than a support ticket.

The dividend as executable, published logic. Takeout rate, division rule and breakage increment expressed as contract code make the dividend the deterministic output of inputs on the ledger, run against rules that were visible before the pool opened, rather than a figure asserted once the pool has closed. "Why $4.30 and not $4.40" stops being a query and becomes a re-execution anyone can perform.

Amendments as new signed transactions. A protest that reopens a pool writes a correction rather than silently overwriting a figure, so the before and after both survive. For a product where post-declaration change is routine, an audit trail that keeps the original is the whole point.

None of this requires the host to surrender control of the pool. It requires the host to publish what it already computes to a substrate the bettor can read. The transparency is the deliverable; the pool operator stays the pool operator.

Where it does not line up

This is the half that keeps it a design essay.

Latency before the jump. Tote pools take money until moments before the start, and the interesting money arrives late. Public-chain finality is measured in seconds to minutes depending on the network; racing's betting window closes on a clock set by the barrier, not the block. A design that needs on-chain settlement of a stake before the gates open is fighting the physics of both systems at once. The workable shape is almost certainly a fast off-chain path for acceptance with the chain as the settlement and audit layer, which means you have reintroduced a trusted component at exactly the moment you were trying to remove one.

Finality versus the settlement deadline. Dividends are expected within a minute or two of correct weight. If the ledger that computes the dividend has not reached the confirmation depth you would trust for a payout in that window, you either pay on an unconfirmed state and accept reorg risk, or you make punters wait for finality and lose the thing that makes the tote feel live. That is a genuine trade-off, not an implementation detail, and it is the same confirmation-depth problem our wallet work runs into, transplanted into a context with a stopwatch on it.

Reorgs against settled money. If a block carrying "pool closed, dividend declared, punters paid" is orphaned, you have paid out on a chain state that no longer exists. Our NFT work can usually afford to wait out that uncertainty because nothing expires. A tote cannot; the race is over and the money has moved. Designing for reorg means either a finality guarantee strong enough that the case is negligible, or a reconciliation path for when it is not — and a reconciliation path is the overnight spreadsheet you were trying to abolish, wearing a different hat.

The host is the incumbent computing authority. This is the constraint that reorders all the others. In the live product, the tote host computes the dividend and we route to it. A chain does not remove that authority; at best it makes the host's computation publicly checkable. Any real deployment sits on top of a host that already closes the pool, applies the deduction and declares — regulated, commingled with others, and under no obligation to publish its internal state to a chain because a downstream platform finds it elegant. A provably-fair pool is not a greenfield build. It is a transparency layer negotiated with an organisation that currently has every commercial reason to keep the pool opaque.

The honest position

The appeal is real and it is specific: pari-mutuel is the wagering product whose fairness is already an accounting identity, and the only thing missing is letting the people in the pool see the pool. A chain is a good fit for stake records, for published division logic, and for amendments that keep their own history.

The obstacles are equally specific, and they are about time and authority rather than cryptography. The betting window and the settlement window are both shorter than the finality a bettor should trust, and the entity that computes the dividend today has neither the obligation nor the incentive to move it on-chain.

So this is where we would start if we started: the chain as an audit and settlement layer beneath a fast acceptance path, confirmation-depth-aware payout logic borrowed straight from the wallet work, and an explicit reconciliation path for reorgs rather than a pretence that finality is instant. And this is where we would stop pretending: none of it is worth building until a tote host is willing to publish pool state, because without that the chain is verifying numbers we typed in, not numbers the pool produced.

We route to tote hosts. We are shipping blockchain systems of our own. We have not joined the two, and we would rather say why the join is hard than imply we have made it.

Where a blockchain earns its place in government systems

Wed, 02 Sep 2026 00:00:00 GMT

A land title changes hands. Two years later someone asks who owned the parcel on a specific date, under which encumbrance, and why the boundary shifted. The current record answers the first question and none of the others, because the system stored the latest state and overwrote the path that produced it. That is the failure that makes people reach for a blockchain, and it is worth being precise about what a chain does and does not fix before reaching.

This is a design position, not a delivery record. We have not built a government blockchain, and this post does not describe one running anywhere. It sets out where we think a chain earns its keep in public systems, where it does not, and what it would take to ship one responsibly. We are writing it from adjacent work we have actually done, which we will mark as we go.

What we have actually done here

Two things are lived, and they bound what we are willing to claim.

We have delivered analytics work for government, through an enterprise partner: a computer-vision system for field monitoring that reached 96% identification accuracy against a set of 20,000+ labelled images and cut manual review workload by roughly 70%, and a civic sentiment classifier that reached 92% accuracy on public feedback at volume. Both shipped. Neither is a blockchain, and neither pretends to be. They tell us how public-sector delivery actually goes: the constraint is rarely the model, it is provenance, retention and who is allowed to see what.

We have also done real blockchain engineering, on our own wallet and settlement work rather than for a government client. That is where our view of the technology's edges comes from: signing, key custody, reconciliation against an off-chain ledger, and the operational reality of a record you cannot edit after the fact.

Everything below reasons from those two. When we move past what we have shipped into what we would build, we say so plainly.

A registry's real asset is provenance, not the chain

Ask what a public registry is actually for and the answer is not "storage". It is a defensible history: who asserted what, when, on whose authority, and what the record looked like on any past date. A title, a licence, a permit, a company filing, a benefit entitlement. The value is the lineage, and the ability to reconstruct the state as of a date without trusting whoever holds the database today.

A chain gives you one part of that: tamper-evidence. Append-only structure with hash linkage means an altered past entry is detectable, and the detection does not depend on the good behaviour of the operator. For a registry that has been quietly edited in the past, that property alone can be the whole reason to move.

But most of the win is provenance discipline the chain does not give you for free. Effective-dating (the record carries the date a fact became true, not just the date it was entered), bitemporal history (what we knew, and when we knew it, tracked separately), lineage back to the authorising instrument, and a schema that refuses to overwrite. You can build every one of those on an ordinary relational store, and if you do not build them, a chain underneath buys you a tamper-evident log of an impoverished record. We have thought hard about effective-dated and bitemporal registry modelling; it is the part that carries the value, and it is boring, and it is where projects underinvest because the chain feels like the exciting half.

So the honest framing is order of operations. Get provenance right first. Then a chain is one credible way to make that history tamper-evident and to remove the single trusted operator. It is not the source of the discipline, and it cannot substitute for it.

Programmable settlement: compelling, and harder than the demo

The second place a chain plausibly earns its place in public systems is settlement: conditional disbursement of money against rules. A grant that releases in tranches when milestones are verified. A refund that fires when a condition clears. A subsidy that flows to eligible recipients without a manual payment run. Expressing the condition and the transfer in the same programmable rail is genuinely attractive, and it is the strongest case for a stablecoin.

We have not built a stablecoin. Treat everything in this section as a capability direction we can reason about, not a product, not a token, and not a financial promise. What our wallet and settlement engineering tells us is that the demo is the easy 20%, and the hard parts are the ones a pilot skips:

Custody. Who holds the keys that can move public money, under what authorisation, with what recovery path when a key is lost or a signer leaves? This is an operational and governance problem long before it is a cryptographic one, and it is where most of the risk actually lives.

Reversibility of mistakes on an irreversible ledger. A duplicate disbursement, a payment to the wrong recipient, a rule that fired on bad input. Ordinary payment systems have chargebacks, clawbacks and reversals because mistakes are normal. An irreversible ledger has to reintroduce reversibility at the application layer through compensating transactions, holds and escrow, which means designing the corrections engine with the same care as the settlement engine. Public money makes this non-negotiable.

Regulatory acceptance. A settlement rail for government funds has to be acceptable to a treasury, an auditor and a regulator before it is acceptable to an engineer. That is a longer road than the technology, and pretending otherwise is how these projects stall after a successful proof of concept.

None of this says do not do it. It says the interesting engineering is custody, correction and acceptance, and a proposal that leads with the token has the priorities backwards.

Where a chain does not earn its place

Plenty of public-system problems get a chain proposed at them and should not.

If there is a single legitimate authority for the record and no need to remove trust in that operator, a chain adds consensus overhead and key-management risk in exchange for a property nobody needed. An internal audit log with proper access control and append-only storage is usually the right answer.

If the data is personal and must be correctable or erasable, an immutable ledger of that data fights data-protection obligations directly. The workable pattern keeps personal data off-chain and anchors only hashes or proofs, which is more architecture, not less.

If the win being claimed is "trust" in the abstract, with no named party whose trust you are trying to remove, the chain is decoration. The question that filters most proposals is simply: whose good behaviour are you no longer forced to assume? If there is no clear answer, keep the database.

What it would take

If we were to build a government registry or settlement system on a chain, this is the shape of the work, and it is deliberately unglamorous:

Provenance and effective-dated modelling done first, and provable on an ordinary store, so the chain is an integrity layer over a correct record rather than a substitute for one.
Personal data off-chain, with only hashes or proofs anchored, so correction and erasure obligations stay satisfiable.
A custody and key-recovery design signed off by the people accountable for the funds or the record, not retrofitted after the pilot.
A corrections and reversal engine treated as a first-class part of settlement, because mistakes are ordinary and the ledger will not forgive them for you.
Parallel running against the incumbent system of record, with the chain authoritative only after its history has been reconciled against the old one for long enough to trust.

That is a multi-year programme with a regulator in the room, not a quarter with a framework. We would rather say that up front than discover it at a go-live.

The honest summary

A public registry's value is a defensible, tamper-evident history with lineage. A chain is one way to get the tamper-evidence, and a good one when you need to remove a single trusted operator; it is not where the lineage comes from, and it will not manufacture provenance discipline you did not build. Programmable settlement on a stablecoin rail is a real and compelling direction for conditional public disbursement, and one we have not built: the technology is ready and the custody, reversibility and regulatory-acceptance questions are the actual project.

We have delivered government analytics. We have engineered wallets and settlement. We have not shipped a government chain or a stablecoin, and this essay is where we would start if asked to.

Containers or serverless: the cloud estate we actually run

Wed, 02 Sep 2026 00:00:00 GMT

On a Saturday afternoon, every wagering brand we run peaks in the same minute. The metro meeting that matters most goes off, and the load on one tenant is the load on all of them: bets, price updates, scratchings and cashouts arriving together, because the punters are all watching the same race. Six days a week the same estate sits at a fraction of that. That traffic shape decided the cloud architecture more than any preference did.

This post is drawn from two systems we build and operate, rather than from a reference architecture. One is a multi-tenant tote and fixed-odds betting platform running on Google Kubernetes Engine (GKE) on Google Cloud Platform. The other was a tipping app, BetrBets, that ran serverless on AWS: Lambda behind API Gateway, with DynamoDB for state. They sit on opposite ends of the container-versus-serverless decision, and both are correct, because they carry different load.

The two shapes

Start with the load, because the load is the whole argument.

The wagering platform has a steady baseline and a sharp, synchronised peak. Steady because there is always live racing somewhere and always a book to run; synchronised because Australian racing concentrates demand: a Saturday metro card pulls every tenant to its peak at once. Tenant demand correlates across brands, so you cannot count on one brand being quiet while another is busy. They are all busy together.

The tipping app had a different shape entirely: near-zero between meetings, a spike around each jump as users opened the app to check tips, then back to near-zero. Event-driven, and idle most of the week.

The wagering platform: a synchronised peak wants a shared cluster

Correlated demand sounds like the case against consolidation — if everyone peaks together, where is the saving? The saving is that a shared cluster provisions headroom once, for the aggregate peak, instead of N times for N separate stacks each carrying its own peak buffer. On GKE, horizontal pod autoscaling adds replicas across the estate for the Saturday window and removes them after. The idle cost the rest of the week is one cluster's worth, not N.

That is the consolidation economics stated in infrastructure terms: one multi-tenant estate instead of a stack per tenant. It is also why tenant onboarding is a Helm operation measured in minutes rather than a new environment build. A new tenant is configuration onto machinery that already exists and already scales.

Containers earn their place here for reasons beyond the peak. The bet path holds long-lived state and long-lived connections: Kafka consumers that must stay subscribed, connection pools into ScyllaDB and Cassandra, in-process caches that are expensive to warm. Those want a process that stays up rather than one summoned per request.

We run this on GKE specifically, meaning managed Kubernetes, and it is worth being plain about what we did not choose. We do not run EC2 as primary compute, and we do not run ECS. Self-managed EC2 would mean building and owning the orchestration we get from Kubernetes for free; ECS would tie the workload definition to one cloud and cost us Helm, which is how tenant onboarding stays a minutes-long operation. The Kubernetes API and the Helm packaging around it are the reason the estate is portable and repeatable, so we chose GKE deliberately rather than falling into it.

Why not serverless for the platform

Serverless would handle the Saturday peak — scaling per request is exactly what it is good at. It is the wrong tool anyway. A cold start in the bet path during a metro card is a latency spike at the worst possible moment. Per-invocation connection churn fights the persistent Kafka and ScyllaDB connections the engine depends on. And a platform with a steady baseline pays for that baseline either way, so the headline serverless saving, scale to zero, is one this workload never gets to collect. You do not scale a book to zero.

The tipping app: event-driven spikes scale to zero

BetrBets was the mirror image. Load arrived in bursts tied to race times and fell away to almost nothing between them. Running that on a cluster means paying to keep capacity warm for six mostly-idle days to serve a few busy hours, plus the operational cost of owning the cluster.

So it ran serverless: Lambda for compute, API Gateway at the edge, DynamoDB for state. Between meetings it cost close to nothing because there was almost nothing to run. The bursts fanned out into concurrent Lambda invocations without a capacity decision on our side. Scaling to zero is a real cost lever, and here it was the right one because the traffic genuinely went to zero.

On AWS the container options were there: ECS, or containers on EC2. We did not take them for the tipping app. A warm cluster to absorb an intermittent, stateless workload is money spent on idle capacity plus an orchestration layer to babysit. The serverless fit removed both.

The factor engine: compute you can put down

The third case sits between the two. Our factor engine, which computes racing probabilities and ratings, runs on dynamic compute-node allocation: it claims nodes to compute a card and releases them when the work is done. The cost lever is the release. Probability computation is bursty and schedulable, so the wrong design is a fixed pool sized for the busiest card and left running through every quiet stretch between race meetings. Scaling the compute down when there is nothing to rate is where the money is saved.

The rule underneath all three

Match the compute model to the traffic shape, and be honest about the shape.

Reach for containers on a shared cluster when there is a steady baseline, when demand across tenants is correlated so pooled headroom pays off, and when the workload holds long-lived state and connections. Reach for serverless when load is genuinely intermittent and stateless, when it falls to zero often enough that scale-to-zero is real money, and when a cold start now and then is acceptable.

Reach for neither on autopilot. Serverless punishes a steady, stateful workload that cares about tail latency, hitting it with cold starts and connection churn. A Kubernetes cluster punishes an intermittent one with idle capacity and standing operational cost. The estate is mixed on purpose: GKE where the load is sustained and synchronised, serverless where it is bursty and disappears. The mistake is standardising on one because it is the one you already run.

Elasticsearch as a projection: global search on a wagering platform

Wed, 02 Sep 2026 00:00:00 GMT

A risk analyst on the operations desk types a partial account number into one box and expects back the player it belongs to, every open bet they hold, the markets those bets touch, and anyone else betting from the same card or device. On a platform running many operator brands at once, that lookup has to reach across bets, wallets, players, markets and tenants, over a window of weeks, and return while the analyst is still looking at the screen.

The store that accepts the bets cannot answer that well. This post is about the search index we run beside it to make queries like that fast, and about the one discipline that keeps the arrangement honest. We run this in production on a multi-tenant tote and fixed-odds platform we build and operate, so what follows is delivered work rather than a design sketch.

Why the system of record can't answer these

The transactional store is tuned for one job: accept a bet and record a wallet movement, correctly, under load, keyed by identity. A wide-column store such as ScyllaDB or Cassandra is excellent at "give me bet 4f9c" or "give me the bets for player P", because those read straight off the partition key. It is poor at "find every bet on this runner, placed in the last fortnight, from an account opened this month, across all tenants". That is an arbitrary predicate over attributes that are not the key.

You can chase each of those queries with another secondary index, but every index you add is more work on the write path that has to accept bets when a market is live. So you keep the transactional store lean and deliberate, and you put the ad-hoc query load somewhere built to carry it. That somewhere, for us, is Elasticsearch.

The index is a projection, never the source of truth

This is the rule that everything else hangs off, so it is worth stating flatly. The index is not the system of record, it is a projection of it. The ledger owns the truth about money and bets. Elasticsearch holds a derived, query-shaped copy.

Three consequences follow, and treating any of them as optional is how teams get burned:

If the index and the ledger disagree, the index is wrong, by definition, and the fix is to rebuild it — never to reconcile the ledger towards it.
No money-affecting write ever lands in the index first. Search is a read model downstream of the transactional write, not a second place bets are created.
Anything a payout or a balance depends on is read from the ledger at the point of decision. Search tells an operator which bet to look at; the authoritative figure comes from the store that settled it.

Hold that line and the index is free to be disposable, denormalised and optimised purely for reading. Blur it and you have built a second source of truth that will eventually contradict the first at the worst possible moment.

Indexing the bet and transaction flow

Bets, wallet movements and market changes already flow through Kafka as the platform's event backbone. Indexing rides that stream: consumers read the domain events and project each one into an Elasticsearch document shaped for the query, not for storage.

The shaping is the point. A bet document carries the player, the market, the tenant, the runner and the current status denormalised onto it, so the analyst's lookup is answered by a single query with no joins. The transactional store keeps those things normalised and separate because that is right for writes; the projection collapses them because that is right for reads. Same facts, two shapes, each fit for its own load.

Out-of-order and duplicate events are a fact of any streamed projection, so each document carries a version derived from the source event, and a later version never loses to an earlier one that arrived after it.

Reindexing, and staying consistent

Because the index is disposable, you must be able to rebuild the whole thing from the system of record on demand. You will need to: mappings change, an analysis chain gets tuned, a projection bug ships, or you simply want a clean copy. The pattern is to build a fresh index alongside the live one and swap an alias over once it has caught up, so readers never see a half-built index.

Consistency between ledger and index is eventual, and the honest engineering is in bounding the lag rather than pretending it is zero. The delay between a wallet movement committing and the projection reflecting it is a number you monitor and alarm on. It also sets a hard rule for the tooling built on top: an operator never makes a settlement or payout decision off the projection. The projection points them at the record; the record gives the number.

Relevance versus exact filters

Operator and compliance tooling is mostly exact filters and ranges: this tenant, this date window, this bet status, this stake band. Those need to be deterministic and auditable, because a compliance query that returns different rows depending on a relevance score is not a compliance query. Filters stay filters.

Relevance earns its place in two narrower spots. One is the free-text global box, where someone types a fragment and wants the most likely matches first. The other is fuzzy identity matching — a misspelt surname, a transposed digit in an account number — where typo tolerance is the difference between finding a colluding account and missing it. Keep the two modes separate in the query layer so that nobody accidentally ranks an exposure report by textual relevance.

Global search across entities

The single box resolves across entity types. A fragment can match a player, a bet reference, a market or a tenant, and the results come back typed and grouped so the analyst sees what each hit is before they click it. Type-ahead runs off the same index.

This is also where the platform's risk cohorts become findable rather than discovered after the fact. Fraud rings, VIPs and bonus hunters share patterns: shared funding instruments, correlated staking, device overlap. A search index that has those attributes denormalised onto player and bet documents turns "who else looks like this account" into a query an analyst can actually run, instead of a report they wait for.

Search analytics, and where search stops

Elasticsearch aggregations carry the light analytical load that sits next to search: facet counts on a result set, a quick breakdown of open bets by market, the distribution behind a filtered view. That is search-based analytics, and it is genuinely useful for operator consoles.

It has a boundary, and naming it is what keeps the architecture clean. Heavy time-series and OLAP aggregation over the full history, meaning long windows, high cardinality and many group-bys, belongs in a columnar store built for it. For that we use Apache Pinot rather than pushing the search cluster into a role it is not shaped for. Three stores, three jobs: the ledger settles, Pinot aggregates history, Elasticsearch finds and summarises.

And search settles nothing. On the wagering platform the index never computes a dividend or a payout; tote dividends are declared by the external host the bet was routed to, and the money is the ledger's. Search finds the bet. It does not pay it.

Where this has run

This is delivered. The search index, the Kafka-fed projection, the reindex-and- alias rebuild and the cross-entity operator search all run in production today on the multi-tenant platform we operate.

None of it is specific to wagering. The shape transfers to any operator console sitting over high-volume records: a lean transactional store, a denormalised search projection kept honest as a read model, and a separate columnar store for heavy aggregation. It fits mining sensor and event logs, insurance claims, trading fills and retail transactions equally well. The domain changes; the discipline that the index is a projection does not.

Kafka as the ingestion spine of a racing data pipeline

Wed, 02 Sep 2026 00:00:00 GMT

A result gets corrected forty minutes after the race. A scratching lands during the window when two of your providers are mid-update and one is not. An odds tick arrives out of order because a provider retransmitted a batch. Every one of these is normal traffic on a racing feed, and every one of them breaks a pipeline that assumed the data arrives once, in order, and stays put.

This is a note on how we built and run the ingestion layer for a racing data pipeline: a service that pulls pre-race fields, live odds and post-race results from several providers, around the clock, and lands them in a store that downstream models and reporting read from. It is delivered work, in production. The specific domain is racing, but the shape of the problem — many sources that disagree, corrections that arrive late, consumers that must not miss any of it — is the same in any real-time domain, so the design reads across.

The setup

A Core Java service does the ingestion. It talks to each provider in that provider's own dialect, on that provider's own schedule, and its single job is to get every message onto a Kafka topic as fast as it arrives. Kafka is the spine. From there the data flows into a Cassandra and ScyllaDB store that feeds model training, live pricing inputs and operational reporting.

We could have wired each provider straight to each consumer. That is the version that looks simpler on the first day and is unmaintainable by the third provider. With N providers and M consumers you are maintaining N×M integrations, every consumer has to understand every provider's quirks, and there is no one place where a late correction can be applied once and seen by everyone. A log in the middle turns that into N producers and M consumers, each of which only has to understand the topic.

Why a log, specifically

A message queue that drops a message once it is consumed would be the wrong tool here, and it is worth being precise about why.

The value of Kafka in this pipeline is that the topic is a retained, ordered record, not a transient buffer. Three properties fall out of that, and each one maps to a failure we would otherwise have to engineer around.

Replay. A consumer that had a bug, or that we add six months from now, can start from an earlier offset and rebuild its own state from the same history everyone else saw. When a model needs retraining on how the data actually arrived, corrections and out-of-order ticks and all, the tape is still there. A queue that discards on read gives you one shot at every message and no way to reconstruct what happened.

Ordering where it counts. Kafka orders messages within a partition. If we partition by the racing entity, the meeting or the race, then everything about one race arrives at every consumer in the same order it was ingested. That ordering is what lets a consumer reason about "the latest state of this race" without cross-checking against a second source of truth.

Back-pressure that does not lose data. Providers do not send evenly. Barrier draws, jump times and results cluster; the quiet stretches are quiet. When a burst arrives faster than a consumer can process it, the consumer falls behind in offset rather than dropping messages. It catches up when the burst passes. The buffering is the retained log itself, which means a slow consumer degrades into lag instead of data loss.

The problem that shapes everything: corrections arrive after the event

The hard part of racing data is not volume. It is that a fact you already recorded can change after you recorded it. A result is declared, then amended after a protest. A runner shown as a starter is scratched late. A provider retransmits a corrected batch because their upstream fixed something.

If your store holds one mutable row per race and you overwrite it each time, you have thrown away the thing you most need later: what you knew, and when you knew it. So the topic design treats a correction as a new event about an existing entity rather than an edit to a past one. The result declaration and the amended result are both messages. Downstream, the store keeps them as an append with effective timing, so "what was the standing result at 15:42" and "what is the result now" are both answerable questions.

This is also why we do not settle correctness at the consumer. If every consumer had to independently decide how to fold an amendment into its view, they would drift apart, and two reports of the same race would eventually disagree. The fold happens once, at ingestion.

Normalisation happens once, at the edge

Providers disagree on the boring things, and the boring things are what cost you. They use different identifiers for the same horse, meeting or race. They timestamp in different zones and with different notions of "now". They model a scratching, a gear change or a track condition differently. One sends a full snapshot; the next sends deltas.

The ingestion service resolves all of that before anything reaches the topic. A horse is mapped to one canonical identity regardless of which provider named it. Times are normalised to one clock. Provider-specific status codes are translated into one vocabulary. By the time a message is on Kafka, it is in our shape, and every consumer downstream is spared from ever learning that provider three exists.

Doing this at the edge, once, is the difference between adding a fourth provider being a contained change in one service, and it being a change that ripples into every model and every report that reads the data.

The payoff downstream: leakage-safe features

There is a specific reason all of this timing discipline matters for the models that read the pipeline. A racing model that learns to predict an outcome must be trained only on information that existed before that outcome was known. If a corrected result, declared after the race, leaks into the features used to predict that same race, the model looks brilliant in backtest and is worthless in production. It has learned to read the answer.

Because the pipeline records what was known and when, feature computation can be pinned to a point in time and can exclude anything stamped after it. The correction is still in the store, because we need it for settlement and reporting, but it is fenced off from any feature that represents a pre-race state. The event-time discipline at ingestion is what makes that fence possible; you cannot enforce it downstream if the upstream already overwrote history.

What this is, and what it is not

This is a working ingestion pipeline we operate, in racing. The claims here are about that: the log-shaped architecture, the correction-safe topic design, normalisation at the edge, and the leakage guard it enables.

It is not a general prescription that Kafka belongs in every pipeline. If you have one provider, few consumers, and no late corrections, a log is machinery you do not need yet. The design earns its place at the point where sources multiply and disagree, and where a fact can change after you first recorded it. Racing hits both early. Trading feeds, sensor telemetry and any multi-source real-time domain hit them too, which is why the pattern moves across without much translation.

Predicting an outcome and staking on it are different problems

Wed, 02 Sep 2026 00:00:00 GMT

A model can name the winning horse in eight races out of ten and still empty the account it was betting from. That surprises people who do not bet, and it surprises no one who runs a book. It is the ordinary consequence of treating a good prediction as if it were a good bet.

We know it from both ends of the same problem. Our racing probability engine produces outcome probabilities across win markets and exotic pools: what it estimates each runner's chance to be, race by race. BetrBets, our own tipping product, took those probabilities and had to answer a different question: given a punter's budget, how much should go on this selection, if anything at all.

Those are two pieces of software because they are two problems. The first estimates what will happen. The second decides what to do about it, and the second is where money is made or lost.

Edge is probability minus price

A probability on its own has no sign. It becomes an edge only when you put it next to a price. If your model says a runner wins 40% of the time and the market is paying $3.00, the bet is positive: the price implies roughly 33% and you think it is 40%. If that same 40% runner is paying $2.20, the identical prediction is now a losing bet, because the price has moved past your estimate. The prediction did not change. The edge did.

This is the first thing that separates a model from a strategy. A model that is accurate but always a step behind the market is accurate and useless. The number that matters is never the probability alone, it is the distance between your probability and the one already baked into the current price. That distance is small and short-lived, and reading it correctly is a different skill from producing the probability in the first place.

Sizing is a separate discipline

Suppose you have a genuine edge. You still have to decide how much to stake, and this is where accurate models bet badly.

Stake too little and a real edge earns almost nothing over the number of bets it takes to show up. Stake too much and variance ruins you before the edge arrives, because even a 60%-favoured outcome loses four times in ten and those losses arrive in runs. The mathematics of this is old and unforgiving: bankroll fraction, the Kelly criterion and its fractional variants, the risk of ruin as a function of edge and bet size. None of it is predictive. It concerns survival and compounding, and it runs on top of whatever the model says.

A model hands you a probability and a confidence. Sizing turns those into a number of dollars, and that translation is a discipline in its own right. Get the prediction perfect and the sizing naive, and you have built something that is right about the world and still goes broke.

What BetrBets did with a probability

BetrBets sat on top of the probability engine and did the part the engine deliberately did not: it sized. A punter set a budget and a risk profile (conservative, moderate or aggressive), and each selection's stake was computed from that budget and the model's confidence in that selection, scaled by the profile. A conservative profile bet smaller and skipped marginal edges; an aggressive one committed more of the budget to higher-confidence selections. A person could override any suggested stake before it was placed.

Three design decisions there carry the whole argument. The stake was a function of confidence rather than a flat amount, so the system bet more when the model was more sure and less when it was not. The risk profile belonged to the punter, not the model, because how much variance a person can tolerate is not something a probability can tell you. And the override existed because sizing is a place for judgement in a way that estimating a probability is not.

The same split runs through trading

The reason this matters beyond racing is that the split is domain-independent. A price target on an instrument is a prediction. It says nothing about how much of the book to put behind it, and that second question is the one that separates desks that survive a bad run from desks that do not. Position sizing, exposure limits and drawdown control are the trading-desk names for what a staking profile does for a punter: they convert a view into an amount, under an explicit tolerance for variance.

We are not offering investment advice and we make no claim about returns. The point is narrower, and it is an engineering point. If you are building a system that acts on predictions, whether they are about horses, prices, fraud or demand, the prediction model and the sizing policy are separate components with separate owners, separate failure modes and separate tests. Collapsing them into one is the most common way a promising model becomes a system that loses money with confidence.

Where this is grounded

This is grounded in delivered work, not a proposal. The racing probability engine and BetrBets are products we built and ran. What this post does not do is quote their results: the subject is the decision discipline the engineering forced on us, not a promise that any staking strategy makes money. A system that stakes real money can lose it, and a better prediction does not remove that. It only changes the odds you are working against.

Real-time operational analytics with Apache Pinot

Wed, 02 Sep 2026 00:00:00 GMT

A tenant operator opens their reporting portal at 2pm on a Saturday and wants one thing: how much money is in, what the current exposure is, and which players are near a risk threshold — as of now, not as of last night's batch. This post is about the store that answers that question, why it is not the same store that accepted the bets, and how the two fit together. It is a lived account: this is the architecture we build and run on a multi-tenant tote and fixed-odds betting platform in production, not a proposal.

The claim this has to support

One of the things the platform does is present a single operational reporting portal, per tenant, showing live money-in, exposure and risk position, so an operator sees the day as it happens instead of reconciling it overnight. That sentence is easy to write and expensive to earn. The cost is almost entirely in where the numbers come from.

The obvious first instinct is to point a dashboard at the database that already holds the bets, wallets and ledger. That database exists, it is authoritative, and it is the wrong thing to query for this.

OLTP and real-time OLAP are different jobs

The transactional store, for us a wide-column store in the Cassandra family (ScyllaDB), is optimised for the write path. It accepts a bet, moves a wallet balance, writes a ledger transaction, and reads a single customer's state back by key. Those are point operations against known partition keys, and the data model is shaped around them: one row, one player, one bet, fetched by id.

A dashboard asks a different shape of question. "Sum money-in across every market for this tenant in the last hour." "Count players whose net position crossed a limit today, grouped by product." "Show the top exposures right now." These are aggregations that sweep many rows, filter on several dimensions, and group. Run them against the transactional store and you are asking a key-value engine to behave like an analytics engine: you scan partitions it was not laid out to scan, you compete with the write path for the same nodes, and the operator who just wanted a number is now a source of latency for the bet that pays for the platform.

A real-time OLAP store is built for exactly that second shape. Columnar layout, so an aggregation reads only the columns it touches instead of whole rows. Inverted and range indexes on the dimensions you filter and group by, so a "where tenant = X and product = Y and time in last hour" narrows before it scans. And crucially, ingestion that keeps the store current within the design target of sub-second freshness rather than a nightly load. We use Apache Pinot for this layer. "Sub-second" here is the category property we design toward, the reason you reach for a store of this class at all, not a benchmarked figure for our deployment, and this post does not quote one.

Event-at-a-time, not batch

The old shape of analytics was a batch: at end of day, extract from the transactional store, transform, load into a warehouse, and report on it in the morning. That is the overnight reconciliation the portal is meant to replace, so loading Pinot the same way would defeat the point.

Instead the platform emits snapshot events onto the event bus (Kafka) as state changes. When a player's position moves, when a domain-level aggregate shifts, an event describing that new state is published. Pinot consumes those events directly from Kafka and makes each one queryable as it arrives, appending to real-time segments and sealing them into immutable ones as they age. There is no scheduled load. The pipeline is: transactional write happens, a snapshot event is produced, Pinot ingests it, the dashboard query sees it. The lag across that path is what "real-time" has to mean, and it is a property of the ingestion design, not of running a batch job more often.

One consequence worth stating plainly: this is a second read model derived from the transactional store, not a replacement for it. The bet, the wallet and the ledger remain authoritative in the OLTP store. Pinot answers "what is the picture across many things right now" and never owns money. If the two ever disagree, the transactional store is right and the analytics store is stale — and designing the event flow so that staleness stays inside the freshness target is most of the actual engineering.

Per-tenant query isolation

The platform is multi-tenant, so a query for one operator must never read another operator's data, and one heavy tenant must not starve the rest. Tenant identity is a first-class dimension on the snapshot events and therefore a first-class, indexed predicate on every query, so a tenant's reporting is scoped by that predicate before anything is scanned, so isolation is a property of the data model rather than something enforced at the application layer after the fact.

Table design carries some of the isolation load too. The responsible-gambling limit data is a concrete case. It began life alongside other player state, and the queries that check limit proximity, which run often and need to stay fast under load, were paying for columns and rows they did not use. We split the responsible-gambling limit tables out into their own tables tuned for those access patterns. The limit-proximity query now reads a narrow, purpose-shaped table instead of contending with everything else about a player. That kind of split is the unglamorous majority of making a real-time store hold its latency as tenants and volume grow: you shape the tables around the questions, and you revisit the shape when a question gets hot.

What this does not solve

A real-time OLAP store is not a system of record and should never be treated as one — it is a derived, eventually-current view, and for anything that must be exactly right at the instant it is read, you go back to the transactional store. It also does not remove the need to design the events well: garbage snapshots in, confidently-aggregated garbage out, faster than before. And it adds an operational surface: a store to run, a Kafka topic set to keep healthy, segment retention to manage, all of it real cost the portal has to be worth. On our platform it is, because the alternative is either a slow dashboard that fights the bet path or an overnight batch that answers yesterday's question.

Where this transfers

Nothing above is specific to wagering. The pattern is: a high-volume transactional system, a need to see aggregate state across it in near real time, and a refusal to make the analytics load fall on the write path. Fraud and anomaly monitoring, live logistics and inventory positions, trading and exposure desks, IoT and telemetry dashboards, retail point-of-sale rollups across many sites — all of them are the same shape. Snapshot the state changes onto a bus, ingest them event-at-a-time into a columnar real-time store, index the dimensions you filter on, and keep tenant identity in the data model. The wagering portal is where we built and run it; the shape is domain-independent, which is the honest reason it is worth writing down.