Schema-migration crash loops at deploy, and the tooling that ends them

Delivered for a the multi-tenant tote and fixed-odds wagering platform we build and operate, through Platform to Platform and 24×7 Operations Desk.

The challenge. Rolling deploys raced additive ScyllaDB schema migrations, so new pods expecting columns the live schema did not yet have crash-looped on startup and held the rollout hostage.

Context

The platform is a multi-tenant tote and fixed-odds wagering platform we built and operate, running as a set of services on Kubernetes with ScyllaDB as the primary store. Racing does not pause for release windows, so deploys are rolling: new pods come up alongside old ones and traffic shifts as they pass health checks.

Most services own tables in ScyllaDB, and most releases that touch a table are additive — a new column to carry a new field. That is the safest category of schema change there is, which is exactly why this failure kept getting past review: nobody treats an added column as a risk.

The problem

A service rolls out carrying code that expects a column the live schema does not have yet. Its startup migration check fails and it logs the signature — Error on Migration, followed by a line naming exactly what is missing. A real captured instance, from our own platform’s logs:

{"level":"ERROR","message":"Missing Columns: [voucherracekey text]"}

The pod exits. Kubernetes restarts it. It exits again. A crash-looping pod takes the whole rollout hostage: the deployment never reaches its ready count, old pods keep serving, and nothing progresses until a person intervenes.

The underlying cause is worth stating plainly, because it applies to any Cassandra- or Scylla-family store: schema changes are not transactional with code deploys. Nothing in the stack enforces an order between DDL applied to the cluster and the rolling deployment of the code that depends on it. When an additive migration races the rollout (applied late, applied to a different environment first, or simply missed in the release sequence), some pods run code expecting columns that do not exist yet.

The operational cost was a paged engineer at deploy time: read the pod logs, recognise the pattern, hand-write the ALTER TABLE, apply it, restart the rollout. The diagnosis was identical every time, and the error message itself named the missing columns and their types. A failure that prints its own remediation is a failure asking to be tooled.

What we built

Remediation built directly into the deploy pipeline. On every deploy, the pipeline:

1. Watches the rollout. A deployment that stops progressing because pods are crash-looping is flagged automatically.
2. Reads the failing pods’ logs and matches the signature — Error on Migration with a Missing Columns line.
3. Extracts the exact missing column and type pairs from the message.
4. Prints the precise ALTER TABLE ... ADD ...; statements it intends to run, verbatim, for the engineer to read.
5. Applies them only on human confirmation, then restarts the rollout so the new pods come up against the corrected schema.

The confirmation step is a design decision, not a limitation. Schema DDL is never auto-applied. A missing column is occasionally the symptom of something else (the wrong build deployed, or a migration intended for a different environment), and auto-applying the ALTER would paper over exactly the cases that matter. An engineer reading the proposed statement catches those in seconds; the tooling’s job is to make that read the only manual step left.

Equally important is what the tooling does not do. It handles this one failure class: additive columns a service expects and the schema lacks. It will not propose dropping a column, changing a type, or touching data, and any migration failure outside the signature still pages a human, deliberately: a failure we have not seen before deserves a person looking at it.

Results

No figures are cleared for publication here, and the story is the tooling itself, so the results are qualitative:

• This class of incident went from a paged engineer diagnosing at deploy time to a confirmed, one-keystroke remediation inside the deploy itself.
• The diagnosis is now encoded. The pipeline shows the exact statements it will run, so review means comparing the proposed DDL against the intended migration instead of reconstructing cluster state from pod logs under deploy pressure.
• Rollouts no longer stall silently. Detection is part of watching the rollout, so the failure surfaces in the deploy channel where the deploying engineer already is.

What we would do differently

Two things, both genuine.

Migrations should have been expand-contract from the start. Apply the additive DDL first and verify it, then ship the code that reads the new column, then remove anything obsolete in a later step. That discipline removes the race entirely; the remediation tooling exists because the discipline was not there from the first table. The tooling is a good seatbelt. We would rather not have needed it, and on new work we sequence migrations this way from the outset.

The detection tooling came later than it should have. The first crash-loop was treated as a one-off release slip rather than a class of failure: fix by hand, move on. But the log signature was already fully machine-readable — column names and types, in a stable format. The rule we took from it: when a failure prints its own remediation in the error message, build the tooling immediately. Waiting for a pattern to prove itself recurring just means paying the manual cost more times than necessary.