· Protocolzone
A tenant operator opens their reporting portal at 2pm on a Saturday and wants one thing: how much money is in, what the current exposure is, and which players are near a risk threshold — as of now, not as of last night’s batch. This post is about the store that answers that question, why it is not the same store that accepted the bets, and how the two fit together. It is a lived account: this is the architecture we build and run on a multi-tenant tote and fixed-odds betting platform in production, not a proposal.
The claim this has to support
One of the things the platform does is present a single operational reporting portal, per tenant, showing live money-in, exposure and risk position, so an operator sees the day as it happens instead of reconciling it overnight. That sentence is easy to write and expensive to earn. The cost is almost entirely in where the numbers come from.
The obvious first instinct is to point a dashboard at the database that already holds the bets, wallets and ledger. That database exists, it is authoritative, and it is the wrong thing to query for this.
OLTP and real-time OLAP are different jobs
The transactional store, for us a wide-column store in the Cassandra family (ScyllaDB), is optimised for the write path. It accepts a bet, moves a wallet balance, writes a ledger transaction, and reads a single customer’s state back by key. Those are point operations against known partition keys, and the data model is shaped around them: one row, one player, one bet, fetched by id.
A dashboard asks a different shape of question. “Sum money-in across every market for this tenant in the last hour.” “Count players whose net position crossed a limit today, grouped by product.” “Show the top exposures right now.” These are aggregations that sweep many rows, filter on several dimensions, and group. Run them against the transactional store and you are asking a key-value engine to behave like an analytics engine: you scan partitions it was not laid out to scan, you compete with the write path for the same nodes, and the operator who just wanted a number is now a source of latency for the bet that pays for the platform.
A real-time OLAP store is built for exactly that second shape. Columnar layout, so an aggregation reads only the columns it touches instead of whole rows. Inverted and range indexes on the dimensions you filter and group by, so a “where tenant = X and product = Y and time in last hour” narrows before it scans. And crucially, ingestion that keeps the store current within the design target of sub-second freshness rather than a nightly load. We use Apache Pinot for this layer. “Sub-second” here is the category property we design toward, the reason you reach for a store of this class at all, not a benchmarked figure for our deployment, and this post does not quote one.
Event-at-a-time, not batch
The old shape of analytics was a batch: at end of day, extract from the transactional store, transform, load into a warehouse, and report on it in the morning. That is the overnight reconciliation the portal is meant to replace, so loading Pinot the same way would defeat the point.
Instead the platform emits snapshot events onto the event bus (Kafka) as state changes. When a player’s position moves, when a domain-level aggregate shifts, an event describing that new state is published. Pinot consumes those events directly from Kafka and makes each one queryable as it arrives, appending to real-time segments and sealing them into immutable ones as they age. There is no scheduled load. The pipeline is: transactional write happens, a snapshot event is produced, Pinot ingests it, the dashboard query sees it. The lag across that path is what “real-time” has to mean, and it is a property of the ingestion design, not of running a batch job more often.
One consequence worth stating plainly: this is a second read model derived from the transactional store, not a replacement for it. The bet, the wallet and the ledger remain authoritative in the OLTP store. Pinot answers “what is the picture across many things right now” and never owns money. If the two ever disagree, the transactional store is right and the analytics store is stale — and designing the event flow so that staleness stays inside the freshness target is most of the actual engineering.
Per-tenant query isolation
The platform is multi-tenant, so a query for one operator must never read another operator’s data, and one heavy tenant must not starve the rest. Tenant identity is a first-class dimension on the snapshot events and therefore a first-class, indexed predicate on every query, so a tenant’s reporting is scoped by that predicate before anything is scanned, so isolation is a property of the data model rather than something enforced at the application layer after the fact.
Table design carries some of the isolation load too. The responsible-gambling limit data is a concrete case. It began life alongside other player state, and the queries that check limit proximity, which run often and need to stay fast under load, were paying for columns and rows they did not use. We split the responsible-gambling limit tables out into their own tables tuned for those access patterns. The limit-proximity query now reads a narrow, purpose-shaped table instead of contending with everything else about a player. That kind of split is the unglamorous majority of making a real-time store hold its latency as tenants and volume grow: you shape the tables around the questions, and you revisit the shape when a question gets hot.
What this does not solve
A real-time OLAP store is not a system of record and should never be treated as one — it is a derived, eventually-current view, and for anything that must be exactly right at the instant it is read, you go back to the transactional store. It also does not remove the need to design the events well: garbage snapshots in, confidently-aggregated garbage out, faster than before. And it adds an operational surface: a store to run, a Kafka topic set to keep healthy, segment retention to manage, all of it real cost the portal has to be worth. On our platform it is, because the alternative is either a slow dashboard that fights the bet path or an overnight batch that answers yesterday’s question.
Where this transfers
Nothing above is specific to wagering. The pattern is: a high-volume transactional system, a need to see aggregate state across it in near real time, and a refusal to make the analytics load fall on the write path. Fraud and anomaly monitoring, live logistics and inventory positions, trading and exposure desks, IoT and telemetry dashboards, retail point-of-sale rollups across many sites — all of them are the same shape. Snapshot the state changes onto a bus, ingest them event-at-a-time into a columnar real-time store, index the dimensions you filter on, and keep tenant identity in the data model. The wagering portal is where we built and run it; the shape is domain-independent, which is the honest reason it is worth writing down.
- apache-pinot
- real-time-analytics
- olap
- kafka
- multi-tenant
- observability