Container memory incidents: connect the kill to the right workload

Preserve the stopped workload's identity first

When a container disappears or restarts, capture its identity before replacing it again. Record the full container identifier, configured component, image identity, start and finish times, restart count and the runtime's reported state. A familiar container name can refer to several successive instances during one incident. A process identifier can also be reused. Preserve the observation time and the selected server with each receipt. This prevents a common investigation error: reading the current healthy replacement and describing its counters as though they explained the container that failed several minutes earlier under a different image or resource configuration.

Start with the user outcome and the change timeline. Did requests fail during a deployment, a report export, a cache warmup or a background job? Did one component restart while siblings continued serving? Keep that context beside the runtime evidence. Docker's inspect reference documents selecting a container object and returning specific fields, which avoids copying an entire environment configuration into a ticket. Prefer a narrow state capture to a broad dump containing credentials and unrelated metadata. The first task is to preserve enough identity to connect subsequent observations to the actual failure without turning diagnostics into another source of private-data exposure.

CONTAINER=reviewed-component  # use the exact ID when preserving an incident
docker inspect --type container \
  --format 'ID={{.Id}} Image={{.Image}} Exit={{.State.ExitCode}} OOM={{.State.OOMKilled}} Started={{.State.StartedAt}} Finished={{.State.FinishedAt}} Restarts={{.RestartCount}}' \
  "$CONTAINER"
# This is a state observation, not a complete termination history.
# The Docker socket is privileged; use approved administrative access.

Interpret exit status alongside the termination evidence

The Bash exit-status convention represents fatal signals as 128 plus the signal number. Under that convention, 137 corresponds to signal nine, SIGKILL. That arithmetic identifies a termination signal, not the actor or resource condition that caused it. An operator, an orchestrator enforcing a stop deadline and an out-of-memory mechanism can all produce a killed process. An application can also choose an exit value itself. Retain the runtime's additional state, event chronology and operating-system evidence before assigning the incident to memory exhaustion. A numerical status becomes useful when it is connected to a specific workload and time.

Read any runtime OOM indication within its documented scope and lifecycle. A current state field is not a complete history of every process in the container or every prior incarnation. Compare finish times with the kernel or service journal available for that interval. Preserve the actual message and identity rather than reducing all events to a red OOM badge. If logs rotated or the original container was already removed, record the evidence gap. The correct conclusion may remain suspected memory-related termination until another source confirms it. Keeping that uncertainty visible is better than changing memory limits for an incident whose real cause was an unrelated forced stop.

Compare the container boundary with the host boundary

A container can encounter its own configured limit while the host still has memory available. Conversely, host-wide pressure can affect a workload whose individual configuration appeared generous. Docker's resource-constraint guide explains container memory controls and the risks of host memory exhaustion. Read the settings actually applied to the affected instance, including the deployment profile that created it. A value in a repository or a provider's server specification is not enough. Keep the container allocation and host capacity in separate fields, then connect both to the workload's observed demand and the other applications sharing that server.

Include inherited constraints in the review. A service manager or parent workload group can impose a boundary above the individual container. Check the relevant hierarchy through authorized host administration rather than assuming that one runtime flag describes the entire effective environment. The Linux cgroup v2 documentation describes hierarchical controls, memory boundaries and accounting. Use the installed kernel's supported interfaces and preserve the reviewed group path. A path guessed from a shortened container identifier can point to nothing or to the wrong scope. The useful result identifies which boundary was observed, what it limited and whether the failed workload belonged to it at that time.

Use event counters with their scope and lifetime

In cgroup v2, memory.events includes hierarchical events, while memory.events.local provides local events. The oom_kill field counts processes in the relevant scope killed by an OOM killer; it does not by itself prove that the leaf group's own limit triggered the kill. Read counter changes across a known interval and correlate them with the workload chronology. A nonzero cumulative value can describe an earlier event. Preserve the group identity and the collection timestamp, because a recreated group can begin with a different counter lifetime. Treat these files as evidence with a defined scope, rather than as a universal container incident log.

Capture the surrounding state before remediation when practical. Read the current allocation and configured boundaries alongside the event counters, then compare a later sample after the intervention. Avoid interpreting a lower current allocation after a process died as proof that memory was never exhausted. Likewise, a high allocation does not establish a leak without a workload and time comparison. Retain missing files and permission failures as explicit limitations. These diagnostics differ across kernels and runtime arrangements, so the operating guide should identify what the installation actually supports. A concise pair of scoped observations can be more informative than a large collection of counters copied without their group paths or reset times.

# Determine the actual group through the runtime and host before replacing this.
CGROUP=/sys/fs/cgroup/REPLACE_WITH_REVIEWED_CGROUP
date -u
for FILE in memory.current memory.max memory.high memory.events memory.events.local; do
  printf '\n%s/%s\n' "$CGROUP" "$FILE"
  if [ -r "$CGROUP/$FILE" ]; then
    cat "$CGROUP/$FILE"
  else
    printf 'Unavailable for this group or permission context\n'
  fi
done
# Preserve the group identity, lifetime and a second timed sample.
# Review relevant ancestors separately; no limits are changed here.

Separate runtime heap measurements from total footprint

An application's heap graph can remain below its configured ceiling while the process or workload uses substantially more memory elsewhere. For a concrete example, Node.js documents memoryUsage fields that distinguish heap values, external allocations and resident memory. Those measurements answer different questions. A container-level observation can also include work beyond one application process. Record the metric name, unit, scope and collection point before comparing two charts. A language-runtime heap limit is not automatically the complete container budget, and a discrepancy between the two views is a reason to inspect their definitions before diagnosing a collector defect.

Use the application's supported profiling tools on an appropriate test or carefully controlled diagnostic path. Heap snapshots and detailed dumps can be expensive and may contain sensitive values, so their collection needs a deliberate operational plan. Begin with lower-cost counters, workload size and a reproducible trigger. If a report export loads a large result set, compare its allocation behavior with a bounded or streamed implementation. If native buffers or child workers are involved, inspect those owners directly. The result should connect memory growth to a component and activity. Repeatedly increasing a heap limit without observing total footprint can move the same failure closer to the container boundary.

Look for pressure before a hard failure

Memory trouble can damage latency before a process is killed. The kernel's pressure stall information guide describes time lost to resource contention and the meaning of its some and full measurements. These signals describe stall behavior, rather than a percentage of capacity allocated. Compare pressure, response latency and useful completion in the same window. A busy allocation chart and a pressure chart can tell different stories, so keep their units visible. When the selected collection does not include pressure information, state that limitation rather than reconstructing stall time from a memory percentage or an isolated user complaint.

Observe the workload around the onset. A growing queue, concurrent report generation or overlapping background workers can increase memory demand even when each individual request looks reasonable. Compare concurrency and input size with the normal operating envelope. A lower concurrency limit can restore useful throughput if the workload was exhausting memory through too much simultaneous work, but it needs an understandable overload policy. Do not replace one failure with unlimited waiting in the frontend. Preserve what was admitted, what completed and what was deferred or rejected. This gives the team evidence for a workload or capacity decision before another hard termination erases the most useful process state.

Budget transient peaks as well as steady state

Capacity planning needs the operations that overlap. Count application workers, cache warmup, backups, report exports and temporary release overlap on the same host. An application's ordinary steady-state footprint can be comfortable while an infrequent transformation briefly needs much more. Preserve those peak conditions in the operating guide. A deployment that starts a replacement before the previous process exits may temporarily run both footprints, depending on its actual procedure. Read the chosen release operation instead of assuming a particular orchestration behavior. The budget should describe the real maximum coexistence that the operator permits, including the host's management and recovery reserve.

Separate a one-time peak from sustained growth. Compare repeated controlled runs with the same input and concurrency, allowing the runtime's normal lifecycle to complete. If demand grows with input size, document the supported bound. If retained allocation grows between equivalent runs, investigate ownership and release behavior through the appropriate tooling. A restart can lower current usage without answering either question. Preserve a compact before-and-after record for the proposed correction. The practical outcome may be streaming, smaller batches, bounded concurrency or a justified allocation increase. Choose the change that explains the observed condition rather than prescribing the same memory number for every service and language runtime.

Restore service without erasing the cause

Immediate recovery and a permanent correction can be separate actions. A supported component restart may be appropriate to restore availability while the owner investigates a leak or workload spike. Record that purpose, retain the failed instance's evidence where the runtime allows it and verify the restored user-facing flow. Avoid an endless restart loop that repeatedly warms caches, replays expensive work and discards diagnostics. A finite restart policy needs an escalation path when recovery does not hold. Track successful service time and recurring failure intervals so a temporarily green process cannot hide the pattern that keeps interrupting customers.

Before replaying interrupted work, inspect its progress and side effects. A killed worker can leave a completed database mutation, a partial file or an external operation awaiting reconciliation. Resume through the job or application's supported recovery semantics. Repeating everything from the beginning can duplicate effects or recreate the same peak allocation immediately. Keep the output destination, operation identity and completion receipt with the recovery decision. If the incident involved a deployment change, use the configured rollback mechanism and account for persistent data compatibility. A memory incident should end with a deliberate service and work recovery, not just another container that happens to be running.

Apply resource changes through the owned profile

A limit change belongs to the component's trusted deployment configuration. Record the selected target, the old and proposed settings, the host reserve and the expected workload. If changing a profile requires an operator-installed configuration update, make that change through the appropriate ownership path before a release. Do not treat an ad hoc runtime edit as a durable fix when the next recreation will restore the old setting. Preserve the resulting runtime receipt and read the applied configuration afterward. The useful question is which boundary the serving instance now has, rather than which file an operator intended to edit on a different machine.

Keep software and capacity decisions explicit. A new image may reduce memory demand without changing a resource limit; a larger allocation may restore a workload without correcting a leak. Label each result according to the evidence. In a manual release environment, building, publishing and applying software remain deliberate operations with their own identities. A repository push does not establish that the corrected implementation is serving. Verify the component after application and watch the same trigger that exposed the original failure. This connects the resource plan to the deployed behavior while keeping any unresolved diagnosis or additional capacity requirement visible to the service owner.

Read Hostwatch evidence at the level it actually covers

Hostwatch provides configured server and component context alongside the resource and service observations collected by that installation. It can help identify the affected placement and compare the incident window with nearby workloads. It does not automatically prove a particular kernel termination cause from a generic memory chart, nor does a current container observation reconstruct a removed instance's history. Preserve the distinction between host, container, runtime heap and event evidence. When a specialized diagnostic was collected through authorized host administration, retain its source and scope beside the Hostwatch observation instead of presenting it as a measurement the product always supplies automatically.

Use this combined record to make the next action specific. Identify the component that changed, the boundary that was observed and the missing evidence that limits the conclusion. If only the host was reachable, state that the container-level explanation remains unresolved. If the source image or deployment revision changed during the incident, preserve both identities. These details let a second operator continue the investigation without guessing which process the original chart described. The fleet view is most useful as a reliable connection between the service and its environment, with specialized memory conclusions grounded in the relevant runtime and operating-system evidence.

Verify the correction against the original trigger

Repeat an appropriate controlled version of the workload that caused the failure, using bounded concurrency and a recovery plan. Compare completion, latency, allocation and relevant counter deltas over the same scope and interval. Check that the application returns resources after work finishes and that siblings retain their expected service quality. A successful empty health endpoint is insufficient when a large export caused the incident. Keep a low-impact production verification separate from any heavier test fixture. The result should demonstrate that the corrected component can complete the intended work within its agreed operating envelope, rather than merely survive a few idle minutes.

Close the incident with the evidence, applied change and remaining limit. Record whether the cause was confirmed or suspected, which old instance supplied the diagnostic and which new image or configuration now serves. Preserve the recovery instructions for interrupted work and the criteria for escalation if the pattern returns. Avoid making an indefinitely larger allocation the only institutional memory of the incident. A useful memory runbook explains workload ownership, transient peaks, effective boundaries and the observations that connect a termination to its cause. That explanation gives the team a repeatable route from the next failed request to a correction it can actually verify.

• Preserve container ID, image, configuration and termination times.
• Correlate runtime state with operating-system evidence before naming the cause.
• Review effective boundaries and counter deltas at the correct group scope.
• Compare heap, process and workload footprints using their actual definitions.
• Verify the original trigger, interrupted-work recovery and sibling service quality.