Route for the result. Respect the budget.

Two routing decisions, different responsibilities

Network routing determines how traffic reaches a service. LLM request routing chooses which eligible model or inference endpoint should process an application task. The two can interact, but a shorter network path does not establish that a model produces a better answer.

Define the task before choosing the routing strategy. Extracting a fixed field, classifying a short message, and drafting a long technical response have different quality and latency requirements. A single “best model” ranking is unlikely to capture all of them.

Create an eligibility policy first

Apply data, capability, and permission requirements before comparing speed or price. A model must support the input type, fit the application’s output contract, and be approved for the information it receives. Keep credentials on the trusted service side of the application.

Begin with a small rule set that is easy to inspect. For example, send a validated extraction task to a model that has passed the task’s evaluation set. Escalate ambiguous cases only when a measured benefit justifies the additional delay and cost.

Evaluate the actual task

Use representative examples, including incomplete or adversarial input. Check required fields, factual grounding where relevant, and how often a response needs another attempt. Record model and policy versions so changes in the output can be investigated.

Treat the request deadline as shared

The application deadline includes time spent waiting, connecting, processing, and recovering. A retry or fallback consumes the remaining budget; it does not restart the user’s clock. Bound the maximum number of attempts and avoid letting multiple layers independently multiply retries.

Gateway features differ. Cloudflare AI Gateway’s request-handling documentation provides a concrete reference for provider timeouts, retries, and fallback configuration. Review the exact semantics of your chosen gateway before treating a setting as an end-to-end deadline.

Plan for partial output

A streamed response changes what recovery means. Once the application has displayed part of a result, switching to a different model can produce duplicated or contradictory text. Decide whether to stop with a recoverable error, resume a supported operation, or clearly start a new attempt.

A tool call also needs a separate authority boundary. A model’s request to perform an action is not proof that the user authorized it. Validate arguments and permissions in application code before any external effect occurs.

Record why a route was chosen

• Log the policy version, task category, selected model, and safe reason code.
• Measure first-response time and complete-response time separately.
• Track success, fallback frequency, validation failures, and cost per useful result.
• Avoid storing raw prompts by default when less sensitive metadata can answer the operational question.

The LLM routing playbook develops this approach step by step. Use the analytics guide to define the evidence, and the AI cost model to account for every attempt rather than just the first request.