Retry and fallback decisions for AI requests
Classify a failed AI request, set a bounded retry budget, reconcile uncertain tool or job outcomes, and choose an approved fallback that preserves the task.
Quick answer
Classify the actual error before retrying. Give the logical task a finite deadline and attempt budget, check whether earlier jobs or tool actions completed, and choose only an approved alternative that meets the original requirements. These are application-design recommendations; IsAnythingDown provides saved status evidence and does not run your requests or recovery actions.
Define success and the deadline first
A retry policy needs a business outcome, not merely a successful connection. For a text task, success may require a complete answer that passes a schema check. For a media task, it may require a usable delivered artifact. For an agent, it may require reconciling tool actions and producing a verified final result. Write that definition and the user's total deadline before choosing delays or attempt counts.
Keep a logical task identifier separate from each upstream attempt. This makes multiple request records explainable without counting one user action several times. Record cancellation and expiration too. A delayed result that arrives after the task is irrelevant can consume money and confuse users even when the underlying API technically succeeded. Reliability includes stopping unnecessary work as well as recovering useful work.
Classify the error using the endpoint's current contract
Read the structured response and operation context before deciding to retry. Invalid input, credentials, permissions, account restrictions, and possible transient failures need different actions. Numeric codes alone are insufficient when provider-specific meanings or operation types differ. Use the current documentation for the actual endpoint and client version. Do not reuse a retry table from another API without checking its assumptions.
The Claude error guide, Gemini troubleshooting guide, and OpenAI error guide illustrate why structured errors and SDK behavior matter. Our recommendation is to preserve that error classification in logs and user-facing handling. A repeated unchanged request cannot repair a deterministic configuration problem, while some transient failures may justify waiting within a finite budget.
Give one layer responsibility for the total budget
Inventory retries in the SDK, gateway, application, queue worker, and user interface. Several layers making only a few attempts can still multiply traffic substantially. Choose a layer that can enforce the logical task's total deadline and attempt budget, or propagate the remaining budget across layers explicitly. Preserve how many upstream attempts already occurred before a fallback decision.
For appropriate transient failures, use increasing delays and jitter to avoid synchronized bursts. Incorporate a valid server wait instruction when applicable. Cap both elapsed time and attempts, and stop on cancellation. Specific delay values should follow the endpoint contract, observed workload, and user deadline rather than being copied as universal constants. An interactive request and an overnight batch usually have different useful waiting periods.
Reconcile uncertain outcomes before replay
A timeout or broken connection can occur after the provider accepted work. Preserve a job or request identifier when available and inspect the existing operation through its documented mechanism. Do not assume that “the client saw an error” means “the server did nothing.” For state-changing tools, inspect the external result before replaying the entire generated interaction.
Use documented idempotency or deduplication mechanisms only where the relevant endpoint supports them. Do not invent a universal header or exactly-once guarantee. Your application may need its own action ledger and reconciliation process. When an outcome remains unknown, present that state honestly instead of automatically converting it into failure and creating a replacement that could duplicate a payment, message, deployment, or generated job.
Treat partial streams as incomplete business results
HTTP response headers and first output are milestones, not a completion guarantee. Record whether the expected completion signal arrived, whether output passed validation, and whether the user saw partial content. Preserve the boundary between the original stream and a replacement response. Silently appending a new attempt can create contradictory text or malformed structured output.
If a stream produced tool actions, reconcile those actions before asking another model to continue. If the result was only text, decide whether to discard, retain as a marked draft, or regenerate it. That decision belongs to the product's task semantics. A transport library cannot determine whether a partial structured result, code patch or bulk edit satisfies the task.
Design fallback before an incident
An alternate route should be approved for data handling, capabilities, output format, tool contracts, cost, and geographic processing requirements. Test representative tasks in advance. Compatible request fields do not establish equivalent model behavior, and matching model names across providers do not establish identical access paths. Preserve which route generated each result so reviewers can understand a material behavior change.
Do not relax privacy restrictions or send confidential context to a personal account merely because the preferred route is unavailable. A fallback may instead be reduced functionality: queue safe non-urgent work, omit an optional feature, use a clearly dated approved cached result, or explain a temporary limitation. If no alternative satisfies the essential contract, stopping is an appropriate defined outcome rather than a policy failure.
Verify output and account for the extra work
After any attempt, validate the actual business result. For code, run the relevant checks. For structured data, validate the schema and required meaning. For research, inspect whether the evidence supports the answer. For media, confirm that the artifact can be retrieved and used. A fallback that returns quickly but fails the task contract has not restored success.
Track logical task success, attempt count, elapsed time, cost, cancellations, and uncertain outcomes using explicitly defined denominators. A retry-heavy system can appear healthy if it counts only final HTTP responses. Keep user-visible delay and incomplete output visible in telemetry. Do not relabel official status history as your own request-success measurement; those sources describe different populations and evidence.
Restore the original route deliberately
An official recovery notice is useful context, but verify a representative operation before restoring normal use. Return demand within your established limits rather than sending every delayed task at once. Expire work that is no longer relevant, preserve unresolved original outcomes, and keep the last meaningful error for support. Record what was tested and when. This produces an auditable task-level recovery without claiming that one successful request proves global availability.
Related service pages
Each entry shows its own collection coverage. A linked service is not necessarily automatically collected.
Editorial review date applies to this guide. Live report collection times appear on the service pages.