TL;DR: Backoff reduces retry frequency; jitter spreads clients that would otherwise retry together. Also bound attempts, elapsed time and aggregate retry work. Retry only failures that may improve and operations whose repeated execution is safe. A timeout leaves the outcome uncertain, and sending an idempotency key helps only when the server actually enforces its contract.
How to approach it
Define which failures the caller may retry and what one logical operation means. Then implement the delay, deadline and shared budget. Use a fake clock and seeded random generator to test behavior without sleeping. Finish by explaining what the small example leaves to the transport and server.
A strong answer
Suppose many clients see a brief endpoint outage. With identical deterministic delays, they can return in synchronized waves. Random delay reduces that coordination, but it does not create backend capacity or guarantee recovery. Retry, Backoff and Idempotency explains the underlying contract; Capacity and Backpressure covers the service's response to excess demand.
Implement a bounded retry policy
The example below is synchronous. Its adapter raises RetryableError only for an eligible transient failure and supplies an already-parsed, nonnegative retry delay where relevant. Other exceptions pass through unchanged. The shared budget earns one retry credit per ten original calls, stores at most two credits and begins with two; it does not refill merely because time passes.
import math
import random
import threading
import time
class RetryableError(Exception):
def __init__(self, message, retry_after_s=0.0):
super().__init__(message)
if not math.isfinite(retry_after_s) or retry_after_s < 0:
raise ValueError("invalid retry delay")
self.retry_after_s = retry_after_s
class RetryDeadline(TimeoutError):
pass
class RetryBudgetExhausted(RuntimeError):
pass
class RetryBudget:
def __init__(self, retries_per=1, requests_per=10, burst=2):
if any(type(x) is not int for x in (retries_per, requests_per, burst)):
raise ValueError("budget parameters must be integers")
if requests_per <= 0 or not 0 <= retries_per <= requests_per or burst < 0:
raise ValueError("invalid budget")
self.earn, self.unit = retries_per, requests_per
self.limit = self.units = burst * requests_per
self.lock = threading.Lock()
def record_request(self):
with self.lock:
self.units = min(self.limit, self.units + self.earn)
def allow_retry(self):
with self.lock:
if self.units < self.unit:
return False
self.units -= self.unit
return True
def call_with_retry(fn, budget, *, retry_safe=False, idempotency_key=None,
max_attempts=5, deadline_s=30.0, base=0.1, cap=10.0,
rng=None, clock=time.monotonic, sleep=time.sleep):
if type(max_attempts) is not int or max_attempts < 1:
raise ValueError("max_attempts must include the initial call")
if not all(math.isfinite(x) for x in (deadline_s, base, cap)):
raise ValueError("timing parameters must be finite")
if deadline_s <= 0 or base <= 0 or cap < base:
raise ValueError("invalid timing parameters")
rng = rng if rng is not None else random.Random()
end, ceiling = clock() + deadline_s, base
budget.record_request()
for attempt in range(max_attempts):
remaining = end - clock()
if remaining <= 0:
raise RetryDeadline("no time for another attempt")
try:
result = fn(timeout_s=remaining, idempotency_key=idempotency_key)
except RetryableError as exc:
if not retry_safe or attempt + 1 == max_attempts:
raise
delay = exc.retry_after_s + rng.uniform(0.0, ceiling)
if delay >= end - clock():
raise RetryDeadline("retry cannot fit before deadline") from exc
if not budget.allow_retry():
raise RetryBudgetExhausted("shared retry credits spent") from exc
sleep(delay)
ceiling = min(cap, ceiling * 2)
else:
if clock() >= end:
raise RetryDeadline("result arrived after deadline")
return result
The function checks the deadline before every call and passes the remaining time to the adapter. It cannot interrupt arbitrary blocking Python code. The transport must honor that timeout, including connection and response phases. If it returns late, the wrapper reports a deadline failure even though the remote action may have completed. Cancellation and response cleanup also belong to the transport.
This budget is shared within one process. Separate processes need an appropriate aggregate policy if they must share a limit. Keep the budget alive across requests; creating a fresh one per request repeatedly grants the startup burst. Synchronizing the counters prevents concurrent callers from spending the same credit.
Know what the limits actually bound
Eight retry-delay ceilings = 0.1 + 0.2 + 0.4 + 0.8 + 1.6 + 3.2 + 6.4 + 10
= 22.7 seconds
Expected full-jitter sum = 22.7 / 2 = 11.35 seconds
These eight delays would follow failures before eight retries, not eight total attempts.
For N original calls, this budget grants at most:
R <= 2 + floor(N / 10) retries, possibly fewer because stored credit is capped
At N = 100: total attempts <= 100 + 12 = 112
sanity: this is a cumulative work bound, not a guarantee of instantaneous request rate.
The average random wait is half the ceiling, but successful completion need not be faster. Retrying too soon can encounter the same failure. A server-requested minimum delay is added before the jitter here; if it cannot fit the caller's deadline, stop rather than shorten it and retry early.
Assume clients fail at the same moment and this retry has an 800 ms ceiling.
Deterministic
- Every client waits 800 ms
Identical delays put this retry back at the same instant.
Full jitter
- Each draws from 0 to 800 ms
Independent draws spread this retry across the window; some can still cluster.
Classify both the error and the operation
A transient connection failure may be eligible, while a malformed request usually needs correction. Status codes such as 502 or 503 do not by themselves prove that an upstream operation had no side effect. Authentication and permission errors need the appropriate credential or access change, not repeated identical calls.
HTTP's idempotency guidance distinguishes safely repeatable semantics from a merely failed response. For application-level deduplication, reuse one key and payload for the logical operation. Verify the server's key scope, retention and atomic handling; a header alone cannot prevent duplicate work. A naturally idempotent operation may not need such a key.
Test the behavior, not real elapsed sleep
These checks run after the implementation:
class Clock:
def __init__(self):
self.now = 0.0
def time(self):
return self.now
def sleep(self, seconds):
self.now += seconds
c, seen = Clock(), []
def eventually_ok(**kw):
seen.append(kw)
if len(seen) < 3:
raise RetryableError("temporary failure")
return "ok"
assert call_with_retry(eventually_ok, RetryBudget(), retry_safe=True,
idempotency_key="job-7", rng=random.Random(0),
clock=c.time, sleep=c.sleep) == "ok"
assert [x["idempotency_key"] for x in seen] == ["job-7"] * 3
assert seen[0]["timeout_s"] > seen[1]["timeout_s"] > seen[2]["timeout_s"]
c = Clock()
def come_back_later(**kw):
raise RetryableError("busy", retry_after_s=2)
try:
call_with_retry(come_back_later, RetryBudget(), retry_safe=True,
deadline_s=1, clock=c.time, sleep=c.sleep)
except RetryDeadline:
assert c.now == 0 # no sleep or early retry against the server's minimum
else:
raise AssertionError("expected deadline failure")
Also check non-retryable failures, retry_safe=False, exhausted credits, invalid inputs, late results and a sleep that overshoots the deadline. A sustained outage may justify temporarily stopping new attempts or rejecting work; a circuit breaker and retries can coexist. Avoid retrying independently at every service layer, which can multiply work beyond this wrapper's budget.
What interviewers probe next
- “Does jitter eliminate the herd?” It reduces synchronization; identical seeds, shared minimum delays and large populations can still create bursts.
- “What if the operation timed out?” Treat the outcome as uncertain and retry only under verified repeatability or deduplication semantics.
- “Can this enforce a hard timeout?” Only cooperatively through a transport that respects the supplied remaining time.
Common mistakes
- Supplying an idempotency key without checking server behavior.
- Calling a lifetime or credit budget a per-second rate limiter.
- Retrying before a server minimum because the local deadline is short.
- Catching every exception as a transient failure.
Key takeaways
- The eight illustrative jitter delays average 11.35 seconds; that is a wait calculation, not a completion-time promise.
- The example permits at most 112 total attempts for 100 original calls, including its initial burst.
- Bound repeated work and preserve the logical operation's identity across attempts.