Two kinds of out-of-memory, and why telling them apart is the fix

Delivered for a a multi-currency ledger service on the multi-tenant platform we build and operate, through 24×7 Operations Desk.

The challenge. A production JVM ledger service experienced both Java heap exhaustion and container memory-limit failures. Each required different evidence and remediation.

Context

The service is a JVM-based multi-currency ledger on Kubernetes. It records money movements and answers queries over them. Large reporting queries can have a very different memory profile from small transaction writes.

The incidents required distinguishing application heap exhaustion from a container exceeding its total memory limit.

Two diagnostic paths

Java heap exhaustion. An OutOfMemoryError: Java heap space indicates that an allocation could not be satisfied from the Java heap. Heap evidence helps identify retained objects and whether a query materialised too many rows.

Container memory-limit failure. Kubernetes can report OOMKilled when a process is killed for exceeding its memory allowance. Total process memory includes native allocations, thread stacks and other overhead as well as the heap. Application logs alone may not explain the termination. Inspect pod status and events together; exit code 137 alone does not establish the cause.

The operational runbook

1. Inspect application logs and the pod’s termination status together before choosing a diagnostic path.
2. On the heap path, inspect retained objects and reporting query shapes. Paging, streaming or bounded result sizes may be needed; increasing the heap does not resolve unbounded work.
3. On the container path, review the memory limit against heap and non-heap usage. Increasing the heap ceiling without reviewing total memory can make the problem worse.
4. Configure heap-dump capture in advance, with an explicit path and enough storage. A kernel kill may prevent the JVM from writing a failure-time dump.

What changed

The service’s memory incidents are triaged through a documented diagnostic branch rather than treated as one generic memory failure. The operational practice includes configured heap-dump capture and reviewing heap settings alongside container limits.

Lessons for batch processing

A migration handler that catches Exception does not catch OutOfMemoryError, which is an Error. That distinction exposed a gap in our batch failure handling. Catching the error alone does not establish safe recovery. A robust migration design also needs bounded work, durable progress and a restart strategy that does not duplicate or omit records.

Heap dumps can be captured proactively from a running JVM as well as configured for heap exhaustion. They cannot reconstruct the heap of a process after it has terminated. Dump capture also needs operational planning for its pause, storage and data-access implications.

The JDK diagnostic command reference describes GC.heap_dump for a running JVM.