Delivered for a the identity-verification pipeline of a multi-tenant platform we build and operate, through Platform to Platform and 24×7 Operations Desk.
The challenge. Verification state lived in three places — our pipeline, the third-party provider, and the truth — and whenever they disagreed, a customer sat stuck at a deposit or withdrawal gate with no automated way out.
Context
On the multi-tenant platform we build and operate, identity verification gates money movement. A customer cannot deposit past a threshold, and cannot withdraw at all, until their identity checks clear. The checks themselves run through a third-party identity-verification provider: our pipeline submits the customer’s documents and details, the provider verifies them, and a status comes back, eventually, and not always by the path we expected.
That gating is the point of KYC. It is also what turns the pipeline into an ongoing operations concern: the integration being built is only the start. A verification that sticks in flight is a customer who cannot move their own money. They do not see “pending upstream webhook”; they see a platform holding their funds. Every stuck row is a support ticket with a regulator-shaped shadow behind it.
The problem
Verification state lives in three places: our pipeline’s record of the customer, the provider’s record of the check, and the truth. The integration’s happy path keeps all three aligned: submit, wait for the callback, update the row, lift the gate. Operating the pipeline for real means dealing with every way they drift apart:
- Expiry drift. The provider expires a verification session; our row still shows it in flight. In one environment we found fourteen rows stuck in an expired state against the provider — fourteen customers whose checks would never complete on their own, and nothing in the system that would ever notice.
- Orphaned rows. Verification records on our side with no live counterpart at the provider at all: the submission failed after the row was written, or a callback was lost. In another environment, reconciliation surfaced 119 of them.
- Upstream replies that break the parser. The provider occasionally returned an empty response to a status poll, and the status parser threw where it should have treated emptiness as a state. An exception in a polling loop is worse than a wrong answer: the row it was checking stays exactly where it was.
- A scheduler that forgot the platform is multi-tenant. The background job that drives verification checks fired across all tenants when it should have run per tenant. Every request path in the platform is tenant-scoped; this one background job was not. That is a recognisable bug class in any multi-tenant system: tenancy enforced rigorously at the API surface and silently absent from the batch layer underneath it.
- Logs that must stay clean. INFO-level logs in an identity pipeline can accumulate provider credentials and customer PII if nothing stops them. Those logs are themselves PII processing: the data-protection obligations that apply to the database apply to every log line describing it, so redaction there is a compliance control, not tidiness.
None of these is exotic. All of them are what an identity integration looks like after it has been in production long enough for the provider, the network and the release history to have each taken a turn at it.
What we built and fixed
Each item above was a real, separately shipped fix on the platform in the same operating period, and together they add up to a standing operational discipline:
- Reconciliation as a first-class job. The fourteen expired rows and the 119 orphans were each cleared with a targeted reconciliation against the provider: read the provider’s state for every in-flight row, and resolve disagreements in a defined direction. The lasting output is the reconciliation path itself, so the next occurrence of either class is a routine job run.
- Per-tenant scheduling. The verification scheduler now runs tenant by tenant, matching the tenancy model the rest of the platform already enforces. One tenant’s verification backlog can no longer set the pace, or the blast radius, for every other tenant’s checks.
- Defensive parsing at the provider boundary. The status parser treats an empty upstream response as a state to handle, and the polling loop survives the provider having a bad minute. The boundary with a third party is exactly where a parser must assume the least.
- Log redaction as standing discipline. Provider credentials and customer PII do not belong in INFO-level logs, and log redaction was tightened after review to keep them out. Auditing log output is rarely a one-and-done exercise; it is a control that gets revisited.
Equally deliberate is what we did not build: no automatic overriding of the provider’s verdict, ever. Reconciliation aligns our record with the provider’s; it does not decide that a customer is verified. The provider’s verdict is the compliance artefact, and a pipeline that can quietly overwrite it has a much worse problem than stuck rows.
Results
No throughput or volume figures are published for this platform, so the results are the two worked examples above and the operational shape they left behind:
- Fourteen expired-state rows and 119 orphaned rows reconciled; more usefully, both stuck-verification classes now have a standing reconciliation path for the next occurrence.
- Verification scheduling respects tenancy, closing a gap between the platform’s request-path isolation and its background jobs.
- The provider integration degrades predictably when the upstream misbehaves, and no longer halts on the first empty reply.
- The pipeline’s logs can be handed to an engineer, a log platform or an auditor without also handing over credentials and customer identity data.
The discipline transfers well beyond wagering, because the shape of the problem is not a wagering shape. AML/KYC onboarding in banking and fintech, licensing and self-exclusion checks in gambling, identity proofing in government service delivery — each is the same machine: an external verifier of record, a local pipeline that must track it, money or entitlements gated on the answer, and logs that fall under the same data-protection regime as the data. Reconciliation paths for every stuck-state class, tenancy-aware background jobs, defensive parsing at the provider boundary and redaction as a standing control are the operating requirements of that machine wherever it runs.
What we would do differently
Schedulers should have been tenant-scoped from day one. The platform’s request paths were built tenant-aware from the start; the background jobs should have inherited that as a hard rule, and did not. The bug class where a batch job ignores tenancy that every API call respects is common precisely because it fails quietly: nothing errors, work just happens at the wrong granularity. On new multi-tenant work we now treat “which tenant is this job running for” as a required parameter of every scheduled job, enforced in the scheduling layer so no job’s author has to remember it.
Log redaction should predate the first provider integration. Stripping credentials and PII from logs after a pipeline is live means trusting that nothing sensitive was written before the rules were in place. Logs outlive intentions: they get shipped to aggregation platforms, copied into tickets and attached to incident reviews, and every copy inherits whatever the original held. For an identity pipeline the redaction rules belong in the logging layer before the first real customer record flows, reviewed with the same seriousness as the schema.