Proxy Retry Budgets: When to Wait and When to Stop

A useful proxy retry policy answers one question: can this authorized job afford another attempt? Retry only a replay-safe operation with a plausible temporary failure, enough time to finish, an unused attempt allowance and an available shared retry token. A delay that has expired is a reason to recheck those conditions, not permission to send automatically.

This guide is for teams running recurring, authorized data collection or regional checks. It provides a conservative decision method and an offline Python planner. All numbers below are hypothetical; they are not IPHTML performance measurements, recommended production defaults or service limits.

1. Decide what kind of failure you have

HTTP status alone does not locate the fault in the client, proxy or destination. Record the response source and task context before changing the proxy configuration. Our planner uses the following deliberately narrow policy.

RFC 6585 defines 429 as rate limiting and permits a Retry-After header. It does not prescribe a single counting scope. A shared target pause is our conservative coordination policy; use the provider's documented scope when available. Without a usable delay or documented recovery rule, require an owner to resolve the pause instead of repeatedly requeuing the job.

The example accepts only GET and HEAD, with explicit operator and access approval. This is narrower than HTTP's full set of idempotent methods. RFC 9110's idempotency rules explain why replay semantics matter. Do not apply this planner to purchases, uploads or resource creation: a timeout does not prove that the original operation had no effect.

2. Set three budgets before the first request

1. Total attempts per job. In this guide, max_attempts=3 means one initial attempt plus at most two retries. Keep the same job identity and counter across worker restarts and queue handoffs.
2. One end-to-end deadline. Measure elapsed time from the original job start, including queueing, connection work, response reading, waiting and validation. Reserve a full next-attempt execution budget plus positive headroom. A socket read timeout is not this deadline; see the latency breakdown guide for measurement boundaries.
3. A shared retry allowance. Define the target scope and accounting window, then consume tokens atomically in one coordinator. A separate allowance in each worker does not enforce a fleet-wide limit. Source rate limits and concurrency limits remain additional gates.

Choose one layer to own retries. Inspect the HTTP library, SDK, queue and application before adding another loop. AWS's retry guidance describes the overload risk of retries at multiple layers. The local planner below cannot inspect or disable those layers for you.

3. Calculate the wait without shortening Retry-After

Retry-After can contain nonnegative integer seconds or an HTTP date. Interpret seconds from response receipt. For a date, compare it with a timezone-aware receipt time; a past date contributes zero additional waiting. Our planner stops on a malformed value instead of guessing.

remaining = job_deadline - elapsed
backoff_upper = min(backoff_cap, base * 2 ** (attempts_used - 1))
jitter_wait = draw_between_0_and_1 * backoff_upper
wait = max(jitter_wait, server_minimum_wait)

Proceed to WAIT_THEN_RECHECK only if:
  attempts_used < max_attempts
  shared_retry_tokens > 0
  wait + next_attempt_budget < remaining

This full-window jitter formula is an editorial policy. The cap limits our own backoff, not the server's requested delay. A real scheduler should preserve a single not_before timestamp. Do not restart the same relative Retry-After countdown on every poll. Use a monotonic clock for elapsed deadlines, keep wall-clock conversion separate and handle clock uncertainty conservatively.

Assume an approved GET returned 503 after the first attempt: deadline 20 seconds, elapsed 4, next-attempt budget 5, base 1, cap 4, draw 0.5 and Retry-After 6. There is a retry token available and the total-attempt cap is 3. The jitter is 0.5 seconds; the wait is 6; the remaining time is 16. Waiting plus execution needs 11 seconds, leaving 5 seconds of headroom. The result is WAIT_THEN_RECHECK.

4. Reproduce the decisions offline

Download the retry-budget Python planner. It uses only the standard library, makes no network requests and never sleeps. Run its synthetic checks and demonstration:

python3 proxy-retry-budget.py --self-test
python3 proxy-retry-budget.py --demo

The bundled demo uses a separate simple fixture: 19 seconds remain, the server date requires 5 seconds of waiting and the next attempt needs 3. It reports WAIT_THEN_RECHECK, a 5-second delay and 11 seconds of slack. To reproduce the worked example above, save this as check_retry.py beside the downloaded file:

import runpy

planner = runpy.run_path("proxy-retry-budget.py")["plan_next_attempt"]
inputs = dict(
    method="GET", outcome=503, attempts_used=1, max_attempts=3,
    elapsed_seconds=4, deadline_seconds=20,
    per_attempt_timeout_seconds=5, remaining_retry_tokens=20,
    retry_allowed=True, access_allowed=True,
    jitter_draw=0.5, base_backoff_seconds=1,
    backoff_cap_seconds=4, retry_after="6",
)
print(planner(**inputs))
inputs["elapsed_seconds"] = 12
print(planner(**inputs))

Run python3 check_retry.py. The first result waits 6 seconds and has 5 seconds of slack; the second stops for insufficient deadline. These are decisions over supplied values, not evidence that a real request can finish in 5 seconds. The tool does not reserve tokens, execute requests, enforce concurrency or cancel work. Any projected token balance is hypothetical until the coordinator commits a reservation.

5. Turn a per-job rule into a workload limit

Suppose a batch contains 100 jobs, each with one initial attempt and a maximum of three total attempts. Without a shared cap, the upper bound is 300 top-level attempts. Give that entire batch 20 retry tokens, with no replenishment during the batch, and the bound becomes 100 + 20 = 120.

That bound requires one shared atomic counter, stable job identities and no hidden retries. It does not bound bytes or money: response sizes, redirects and browser subrequests can add work. Define the counted request unit, account for those additional requests separately, and add byte or spending controls where required. Use the effective-result cost model to evaluate the observed expense rather than assigning every attempt the same assumed price.

A 429 defer queue also needs a maximum job age and an explicit owner. Whether you charge a defer against a retry counter is a local design choice; resetting the deadline or silently creating a new job would defeat the budget. An expired job should end with a recorded reason, not return as a fresh task indefinitely.

6. Check the integration before increasing volume

• At wake-up, recheck permission, the shared pause, rate and concurrency limits, deadline and attempts; atomically reserve a retry token immediately before dispatch.
• Use an executor that can enforce the whole attempt budget, including validation and cleanup. AWS's timeout discussion explains why individual connection or socket timeout settings may leave work outside their scope.
• Keep a redacted job ID, attempt number, outcome, response receipt time, not-before time, deadline and final stop reason. Do not log proxy passwords, authorization headers or personal page contents.
• On a small authorized sample, compare useful jobs recovered with extra requests, deadline misses and cost. Use the same reliability scorecard definitions before and after; do not change the success rule to improve the result.

If failures remain concentrated on permission, parsing or an unsupported source, a documented API, licensed feed or corrected workflow may be more appropriate than additional proxy capacity. If the job needs a consistent exit, preserve its session requirement when designing recovery.

Next step: once the failed-job evidence is clear, compare the requirements with the rotating proxy product and use the current contact options to confirm applicable limits and session behavior. Provide a redacted time window, affected regions, attempt counts and stop reasons. Buy or expand only after an authorized workload meets your own acceptance criteria; this offline planner establishes no IPHTML speed, capacity or support-response guarantee.