LLM request routing is the policy that decides which model should handle a task and what to do when that choice cannot finish successfully. Its purpose is to meet a defined quality standard within an acceptable wait and budget. Choosing the lowest advertised request price is only one small part of that problem.
Consider an internal writing assistant with three jobs: extract invoice fields, summarize a project discussion, and draft a customer reply. Those jobs need different context, output checks, and error handling. A useful router recognizes those differences and makes its decisions observable. The following framework is an engineering proposal, rather than a claim that any provider guarantees a particular outcome.
1. Establish which models are eligible
Start with a capability registry. For each candidate, record the supported input types, usable context budget, output schema requirements, tool interface, permitted data destinations, and the task evaluations it has passed. Treat model identifiers and provider configuration as versioned dependencies. A familiar name does not remove the need to test changes.
Apply hard constraints first. An invoice containing restricted account information may only use approved endpoints. A task requiring image input needs an eligible image path. A request expecting a structured result needs a parser and validation strategy that work with the selected model. Remove incompatible candidates before comparing speed or cost.
Our AI routing architecture guide distinguishes selection policy from the network path. An API gateway or CDN can forward a model request, apply rules, or manage connections. The model's execution occurs at its inference endpoint unless an explicit runtime is deployed at that gateway location. Do not infer execution placement from the presence of an edge network.
2. Begin with understandable routing rules
A small rule set is easier to audit than an elaborate scoring system with untested inputs. Route routine extraction to the least expensive eligible candidate that passes the extraction evaluation. Route a discussion with a long context to candidates tested on that context size. Reserve a stronger evaluated option for drafting tasks where errors are costly to review.
Keep the language precise: a candidate passes a task evaluation; it is not universally the best model. Document why each branch exists and what evidence would justify changing it. If a classifier predicts task difficulty, include the classifier's own latency, mistakes, and cost in the policy evaluation.
Prefer application metadata over instructions embedded in user content. Task type, authenticated tenant policy, and requested output format are more dependable routing inputs than text that says to use a particular endpoint. The router should enforce allowed destinations even when a document contains instructions that conflict with the application's rules.
3. Budget the whole request deadline
Define a user-facing deadline from request acceptance to the required completion state. For an interactive draft, that may mean a complete response. For a streaming interface, define both time to first useful text and time to completion. A quick opening token does not establish that the remaining answer will arrive promptly.
Reserve time for work around inference
Subtract authentication, retrieval, routing, validation, and delivery from the total budget before allocating time to model attempts. An illustrative eight-second product target cannot support two separate eight-second attempts. Give each attempt only the remaining budget after reserving enough time to validate and return a useful result or an honest failure message.
Check the exact meaning of provider controls. Cloudflare's AI Gateway request-handling documentation describes a timeout based on the first part of the response. That setting alone is not an overall completion deadline. Enforce the application's own total deadline and verify cancellation behavior across the entire request path, including streaming connections.
4. Bound retries and fallbacks together
Separate another attempt from another model
A retry repeats a request against the same candidate. A fallback selects another eligible candidate. They solve different problems: a brief connection failure may justify a retry, while a capability limitation requires a different route or a clear refusal to proceed. Classify errors before deciding what another attempt could accomplish.
Use one shared attempt budget. Otherwise, retries in the client, gateway, and provider wrapper can multiply without the application realizing it. Apply bounded backoff with jitter for appropriate transient failures, honor usable retry guidance, and stop when the remaining deadline cannot accommodate a useful attempt. Do not repeatedly retry malformed requests or missing permissions.
Track a request identifier across all attempts. Generation retries may still consume additional resources, and cancellation does not necessarily undo completed upstream work. For tool actions such as creating an order, enforce idempotency at the action service. A repeated model call must not silently repeat a business action. Validate fallback outputs against the same task requirements as the original route.
5. Define what happens after streaming starts
Streaming changes recovery because the user has already seen part of the result. Before any output is shown, a fallback can often replace a failed attempt cleanly. After output begins, appending a different model's continuation risks a response with inconsistent reasoning, wording, or structure. Make this transition a deliberate interface decision.
For the writing assistant, one policy is to preserve the partial draft, label it incomplete, and offer a fresh attempt. Another is to buffer until a small coherent unit is available, accepting the additional delay. Choose according to the task, then test interruption at several points in the stream.
Measure first useful content, complete output, and successful validation separately. Handle a user cancellation as a state the router understands, with no new fallback attempts afterward. Bound buffering and close abandoned connections. These details determine whether a streaming application behaves predictably when the network fails midway through a response.
6. Evaluate successful outcomes and their full cost
Build an evaluation set from representative tasks with appropriate privacy controls. Include common inputs, difficult cases, long contexts, empty fields, conflicting instructions, and deliberately interrupted requests. Define acceptance before comparing candidates. Invoice extraction can use field correctness and schema checks; drafting needs a rubric covering factual support, required content, and review effort.
Report task success by route and segment. Also record validation failures, fallback frequency, total attempts, time to completion, and user abandonment. A low price per model call can coexist with a high cost per accepted task if the application repeatedly escalates or regenerates responses.
Our AI infrastructure cost guide develops that accounting approach. Include retrieval, routing, model attempts, observability, and relevant infrastructure charges. Use the analytics planning guide to define event boundaries and denominators so the dashboard compares complete interactions rather than an inconsistent mixture of attempts and sessions.
7. Release policy changes as experiments
Version the routing rules alongside the capability registry and evaluation results. Start a new policy with a small representative cohort and a defined rollback threshold. Compare against the prior policy on the same task categories, while accounting for changes in input length, user population, and provider conditions.
Historical traffic cannot reveal how an unchosen model would have answered every request. Where permitted, use a controlled evaluation sample or a carefully budgeted shadow comparison. Avoid duplicating sensitive inputs into unapproved services just to obtain more data. Keep evaluation calls distinct from user-visible attempts in cost and latency reporting.
Record an explanation code for each selection, such as context requirement, task rule, or provider failure. This makes unexpected traffic shifts easier to investigate without storing full prompts in every log. Review sudden changes in those codes before adjusting thresholds; a broken capability detector can resemble a genuine change in user demand.
The CDN analytics guide explains how operational measurements can support decisions without overclaiming causality. Review routing changes whenever a model, prompt, retrieval pipeline, or output validator changes, because each can alter which candidate is appropriate.
Conclusion: optimize a policy you can inspect
A dependable LLM router starts with eligibility, applies understandable selection rules, and spends a bounded budget on useful attempts. Its measurements connect quality, time, and cost to the same completed task. Keep fallback behavior visible, treat streaming as its own recovery problem, and require evidence before expanding complexity. The goal is a policy your team can explain during normal operation and diagnose when a request fails.
