When retries become the outage: contain retry storms and cache stampedes

Count attempts as work, not as new users

A service can receive much more work without gaining a comparable number of genuine visitors. One user action can trigger a client retry, an API retry, a proxy attempt and a repeated background job. If those layers react independently to a slow dependency, the extra work arrives precisely when the dependency has the least spare capacity. The resulting traffic may resemble an external flood even though the immediate amplification comes from your own software. Start the investigation by separating user arrivals, request attempts and completed useful actions. A request counter alone cannot tell which of these quantities increased or whether repeated attempts can safely produce the same business result.

The Amazon Builders' Library article on timeouts, retries and jitter explains why retry behavior needs deliberate design around failures and resource limits. The diagnostic procedure here is our own application of that principle. It is useful during a dependency outage, a cache expiry wave or an ordinary deployment that makes many clients reconnect. The goal is to reduce avoidable attempts while preserving a path for useful work. Increasing every timeout or restarting every component can hide the pattern briefly without changing the amplification rule that created it. Measure that rule before choosing the next intervention.

Draw every layer that can repeat the request

Follow one representative user action from its caller to the final dependency. Mark the layers that can retry, their maximum attempts, deadlines and cancellation behavior. Include library defaults and infrastructure behavior rather than documenting only the retry loop visible in application code. A reverse proxy, SDK and job consumer can each introduce another attempt under different conditions. Some retries occur after connection failure; others happen after a timeout during work that may already have completed. These cases have different implications for resource use and duplicate effects. The map should explain what the system can actually do when the dependency is slow, not merely what the happy-path diagram shows.

Use a simple hypothetical calculation to make the problem concrete. If a caller allows three attempts including the initial call, and the next layer independently allows three downstream attempts for each one, one user action can produce nine downstream attempts. This is an illustration of multiplicative policy, not an observed production measurement. Adding another repeating layer can increase the work further. Record whether each configuration expresses total attempts or additional retries, because those words are often confused. Then identify which layer should own the retry budget for this operation. A clear ownership decision is more useful than tuning several unrelated retry counts until a test happens to pass.

Establish a deadline for the whole operation

A timeout should fit the useful lifetime of the user action. Separate connection establishment, response waiting and the total end-to-end budget where the client library supports them. A short socket timeout may not cover every earlier step, and a long downstream timeout may outlive the caller that could use the answer. Inspect the library's documented behavior rather than assuming one option bounds the whole request. Then compare the deadline with actual dependency latency, including tail behavior during pressure. The right value depends on the service contract and recovery strategy; copying a timeout from another endpoint does not establish that contract.

Cancellation also needs a real path. A caller giving up does not necessarily stop work already accepted by a database, worker or remote service. Check whether cancellation propagates and what happens to work that cannot be cancelled safely. Keep the difference between a timed-out acknowledgment and a known failed operation visible. For example, a payment or account mutation may have completed before the response was lost. Blind repetition can create a duplicate effect even if the network error looks transient. Use the operation's supported idempotency mechanism and investigate uncertain outcomes rather than interpreting every timeout as permission to start the same business action again.

Choose failures that are actually retryable

Not every error deserves another attempt. A malformed request, missing authorization or unsupported operation generally needs a corrected input or access decision. A transient connection problem may justify a bounded retry for a safe operation, while a timeout after a mutation requires stronger evidence about idempotency. Define the policy by operation and failure class. Respect a documented retry instruction when the service supplies one, within the caller's remaining deadline and budget. Do not turn a shared default such as retry every non-success response into an invisible source of traffic across all of the application's endpoints.

Record retry decisions in bounded telemetry that connects attempts to the originating operation. Useful fields include attempt number, failure class, chosen delay and the terminal result, with a request or operation identity appropriate to the privacy policy. Avoid logging full credentials, raw bodies or private query values just to correlate work. During an incident, compare the rate of original operations with downstream attempts and successful completions. If attempts rise while completions fall, the system may be spending capacity on work that cannot improve the user outcome. That observation supports reducing amplification before adding more concurrency to the same struggling dependency.

Back off, add jitter and keep a finite budget

Waiting between attempts gives a dependency room to recover. Randomizing those waits can prevent many clients from waking at the same instant after a common failure. Neither technique removes the need for a cap. Set a finite number of attempts, a bounded delay and an end-to-end deadline. Decide what the caller does when the budget is exhausted: return an understandable temporary failure, offer an approved degraded result or defer eligible work. An unbounded retry loop can keep the incident alive long after the original event ended, and an infinite waiting queue can move the same problem into memory or connection exhaustion.

The pseudocode example later in this article is a policy sketch, not a complete network client. Its important properties are that a single layer owns retries, every attempt consumes a token, delays are bounded by the remaining deadline and uncertain mutations do not enter the generic retry path. Production code should use the relevant library's supported timeout and cancellation features. Test failure behavior against an isolated dependency or fixture rather than sending an uncontrolled wave at the live service. The result to measure is useful completion and downstream load under the chosen failure, not simply whether a request eventually succeeded after enough repetition.

deadline = monotonic_now() + operation_budget
max_attempts = 3                  # includes the initial attempt

for attempt in 1..max_attempts:
    remaining = deadline - monotonic_now()
    if remaining <= 0: return DEADLINE_EXCEEDED
    result = call(timeout=min(per_attempt_limit, remaining))
    if result.success: return result
    if result.mutation_outcome_unknown: return RECONCILE
    if not safe_and_retryable(result): return result
    if attempt == max_attempts: return RETRY_BUDGET_EXHAUSTED

    delay = full_jitter(min(backoff_cap, base_delay * 2**(attempt-1)))
    delay = max(delay, valid_retry_after_or_zero(result))
    if delay >= deadline - monotonic_now(): return DEADLINE_EXCEEDED
    sleep_cancellable(delay)

# Only this layer retries. Calls must honor their timeout/cancellation.
# Review idempotency, concurrency and library behavior separately.

Bound concurrency and queueing before memory becomes the queue

Retries are only one way load grows. A component can accept more concurrent work than its dependencies can serve, retaining connections, request objects and intermediate results while callers wait. A bounded queue and admission policy make the overload state explicit. Choose the maximum in-flight work from the service's actual bottleneck and budget, then define what happens when that limit is reached. A queue can absorb a brief mismatch, but it cannot create processing capacity. If arrivals remain above service capacity, waiting time and backlog continue growing until some other resource or deadline becomes the accidental limit.

Measure admitted work, rejected or deferred work, queue depth, in-flight operations and useful completions together. A falling error rate can be misleading if every request now waits beyond its useful lifetime. Likewise, a fast rejection can be preferable to consuming resources for work that cannot finish, provided the caller handles it with the intended bounded policy. Keep fairness and operation cost in view. A small expensive job class may need a separate budget from cheap cached reads. Treat per-account or subscriber exceptions as application-verified identity decisions, not as assumptions derived from a shared IP address or an untrusted header.

A cache miss is a coordination problem

When a popular cache entry expires, many requests can discover the miss before the first refill completes. If each request recomputes the same expensive result, the cache can amplify load at the exact moment it is meant to reduce it. Coordinate regeneration for the same logical key and bound how long other callers wait. Nginx documents cache locking for this purpose, with explicit timeout and age controls. The same design question exists in application caches: how many independent computations may populate this key while the dependency is slow, and what result can waiting callers legitimately receive?

Choose cache semantics before enabling a convenient proxy setting. A public catalogue can have different freshness and authorization requirements from an account page. An application may allow a short stale response during an upstream failure, or it may require a current answer. Never cache private results under a key shared by unrelated users, and do not ignore security-relevant response headers merely to improve the hit rate. Test cold-cache and simultaneous-expiry behavior in addition to warm-cache performance. A benchmark that begins after every important key is populated can completely miss the regeneration pressure that appears immediately after a deployment or cache restart.

Make the cache key and bypass conditions explainable

The key determines which requests are treated as equivalent. Include every input that changes the permitted response, and deliberately exclude only inputs proven irrelevant. Query variation can produce a large set of keys, while over-aggressive normalization can return the wrong result. Authentication, cookies, language and application-specific state need explicit treatment. For a public-read proxy example, conservatively bypass requests carrying authorization or cookies and preserve the backend's cache-control decisions. The sample configuration is limited to a designated anonymous public location. It is not a template for caching every route on a server, and its storage path and zone need operator preparation.

Observe cache hit, miss, stale and bypass outcomes beside origin attempts and latency. A low hit rate may be correct when most requests are personalized; a high hit rate can still hide bad behavior for the subset of costly misses. Inspect key cardinality and refill concurrency where collection supports them. Do not publish sensitive key values from raw logs just to illustrate a pattern. A useful investigation can compare normalized classes and bounded examples. The practical question is whether the cache reduces the dependency work for requests that are truly equivalent while preserving the application's authorization and freshness contract for requests that are not.

Recovery traffic needs a budget too

When a dependency returns, clients reconnect, workers resume and caches refill. This recovery wave can be large enough to cause another failure. Stagger eligible scheduled work, keep retry jitter in place and consider gradual admission rather than releasing every queued operation at once. Avoid restarting all healthy instances merely because one component recovered; that can discard warm connections and cached state simultaneously. Treat warm-up as a measurable phase. Watch useful completions, downstream connection pressure and the age of queued work while increasing load within the service's real capacity.

Decide what to do with work whose deadline has already passed. Replaying an obsolete interactive request may consume resources without helping its user. Some durable business jobs must still complete, but their idempotency, ordering and duplicate handling belong to the job contract. Preserve the distinction between deferred useful work and stale retries from a caller that has moved on. A service can show fewer immediate errors while silently accumulating a backlog that makes recovery much longer. The end of the original failure is therefore a checkpoint, not automatic proof that the system has returned to its normal operating state.

Use Hostwatch to connect pressure to the right component

Begin with current service placement and the important endpoint's observed host. Compare request outcomes, host pressure, runtime identity and the recent release timeline in a common window. Hostwatch can expose configured service and traffic evidence, while application-level attempt identities and queue internals require collection from the application or dependency that owns them. Do not claim those details were measured if the current integration does not provide them. A bounded list of recent requests can suggest repeated routes; it cannot by itself reconstruct every user's retry tree or prove that an internal retry policy caused the entire incident.

If a correction requires a software release, select the exact component and follow the explicit source, publication and deployment stages. A staged build is not a live fix, and a container restoration does not reverse an application's data mutation. Keep the previous image and application compatibility requirements available. Hostwatch does not install a Git-triggered deployment pipeline for these operations. The useful connection is between the diagnosed amplification rule, the reviewed code or configuration change and the post-release evidence that fewer attempts now produce more useful work. That explanation is stronger than a general claim that another restart solved the spike.

Verify the budget under a controlled failure

Build a small isolated test that makes the dependency slow or temporarily unavailable and counts attempts without involving real customer transactions. Confirm the end-to-end deadline, retry limit, jitter distribution, queue bound and duplicate handling appropriate to the operation. Check what the caller sees when the budget is exhausted. For a cache, test a cold key with concurrent callers and confirm that refill work remains bounded. Include an ordinary authorized flow and a personalized response in verification so that an optimization does not accidentally change access control or freshness. Keep the fixture and expected result with the operating guide for future changes.

Finish the incident with a policy statement that another engineer can understand: which layer retries, which operations qualify, what the total deadline is, how much concurrency is admitted and what happens during cache refill or recovery. Preserve the measured comparison with the previous behavior and any collection limits. The objective is not to eliminate every transient error. It is to keep failures from multiplying into more work than the service can afford, while allowing safe operations a reasonable chance to recover. When those boundaries are explicit, a partial dependency failure has a better chance of staying partial instead of becoming an outage created by the response to it.

• Identify every layer that retries and choose one owner.
• Set total time, per-attempt limits, maximum attempts and idempotency policy.
• Bound concurrency, waiting and refill coordination.
• Exercise a controlled failure and observe downstream attempts.
• Compare useful completions during failure and recovery.