When a production workflow that uses proxies stops delivering usable results, a good response protects the business decision first: who can change what, which work must pause, and when is it safe to resume? This runbook is for an operator coordinating an incident across application, network and business teams. Adapt the thresholds to your own service and permissions before an incident.
1. Declare the incident and assign control
Use a pre-agreed business trigger, such as an important report missing its delivery window, to declare an incident. Record the affected workflow, target group, region, time window and last known good observation. Mark the cause as unknown until evidence supports it. A proxy error alone does not establish which party caused the incident.
Appoint an incident commander to make priorities and handoffs explicit, an operations owner to authorize and execute changes, and a communications owner to publish updates. In a small team one person may hold multiple roles, but the current owner of each must be recorded. Use one shared incident log and agree on the next update time. Google’s incident response guidance describes these roles and the value of a working record.
2. Contain the affected work before changing routes
| Observed condition | Immediate decision | Resume only when |
|---|---|---|
| 403, 429 or an access challenge on a target | Stop the affected requests; do not rotate identities to evade the restriction. Coordinate any target-wide pause and applicable Retry-After. | Access is permitted and the pause and applicable limits have been checked. |
| 407 or unknown authentication state | Stop the affected route and have its owner inspect configuration securely. | Credentials and authorization have been checked without exposing them in the incident log. |
| Impact or cause unknown | Pause nonessential changes and preserve a redacted observation; keep unaffected work within its existing limits. | The commander has an owner and a bounded next action. |
| Fallback is unapproved or its safety is unknown | Hold affected jobs instead of switching broadly. | A specifically approved alternative satisfies access, data and business-state constraints. |
Keep the status codes and time windows, but do not paste credentials, cookies, raw customer data or sensitive URLs into a shared record. For timing evidence use the separate curl latency breakdown; it does not decide incident responsibility.
3. Make one controlled change at a time
The operations owner records the proposed change, affected scope, reason, approver, expected observation, stop condition and a safe rollback procedure before execution. Record a non-secret reference to the previous configuration, not its contents. Pause unrelated deployments; if an urgent safety action cannot wait, record it immediately and notify the commander. Coordinate changes by dependent teams so two responders do not unknowingly alter the same path at once.
Rollback is an option only if its state and business consequences are understood and it is authorized; an old configuration is not automatically safe. Record the actual result and time, then decide the next action. An ownership transfer is complete when the incoming owner acknowledges it and understands the active changes and limits.
4. Restore in stages with explicit stop conditions
Hold: identify the affected scope and confirm access permission, any target-wide pause, rate and concurrency limits, the responsible operator and an approved test plan. Unknown permission or an active restriction means remain on hold.
Canary: run a small, explicitly authorized set of representative jobs within those limits. Check the actual business result, target context and side effects, not just an HTTP status. For a fictional read-only workflow, a team might choose two distinct jobs in sequence; both are illustrative, not a universal acceptance threshold or proof of sustained reliability. Stop on access denial, invalid results or unexpected effects.
Ramp: the owner records the observation window and acceptance criteria before each small increase. Keep new traffic and backlog release within the same applicable shared limits. If symptoms return or the limits are unclear, pause and reassess; do not automatically switch routes or replay more work. Use the reliability scorecard to define job outcomes and the usable-capacity guide to plan a separate controlled capacity test.
5. Triage the backlog before releasing it
Inventory affected job IDs and their business deadlines, last known state and owners. Assign each job exactly one current disposition using this precedence: any uncertain side effect or prior completion goes on hold for reconciliation first, even if the job is expired or suspected to be a duplicate. After reconciliation, classify it as expired/duplicate or eligible as appropriate. Expiry decides whether to replay; it does not remove the need to investigate prior effects. Retain the record even when a job expires. AWS discusses why queued work can become stale and why backlog recovery needs deliberate design in its queue-backlog guidance.
| Job state | Disposition | Release gate |
|---|---|---|
| Reconciled; deadline passed or confirmed duplicate | Mark expired or duplicate; retain evidence and notify the business owner. | Do not replay automatically. |
| Side effect or prior completion uncertain | Hold for reconciliation with the system of record. | Release only after the owner confirms it is safe and still needed. |
| Still useful, authorized and reconciled | Eligible for a bounded release, ordered by agreed business priority and deadlines. | Recheck limits, dependencies, freshness and incident stop conditions for each batch. |
Example only: of 100 hypothetical jobs, 30 are reconciled and expired, 10 have uncertain side effects and 60 are reconciled and potentially eligible. These three groups are assumed to be disjoint; uncertainty takes precedence over expiry. The 60 still need the release checks above; none is automatically replayed. Agree how much shared capacity remains for new work, and stop release if access restrictions return, business deadlines for new work are missed, or duplicate effects or invalid results appear. Recheck the retry owner and shared budget using the retry-budget guide; a backlog is not permission to retry indefinitely.
6. Hand off, close and rehearse
Before closing, the commander records which workflows were observed working, the observation window, unresolved impact, backlog counts and owners for held work. Verify the ledger reconciles and that stakeholders know the next update; a single successful response does not establish full recovery. Record root cause as unknown if it remains unknown. Assign follow-up actions and dates separately from the decision to restore service.
Use the bilingual redacted incident and drill template to practice a handoff, one change and a backlog decision before a real event. Never include secrets or personal data in the shared copy.
If your workflow needs an authorized rotating proxy route, review rotating proxies and contact the team with your requirements and constraints. Confirm suitability and permissions before changing a production route.