AI infrastructure costs: budget for the full request path

AI infrastructure costs are easiest to underestimate when the budget begins and ends with a model's token rate. A user request also crosses delivery services, executes application code, reads data, creates telemetry, and sometimes runs again after a failure. The useful budget follows that entire path. It also names what the system is trying to accomplish, because an inexpensive response that misses the user's need can make the overall service more expensive.

1. Draw the request path before opening a spreadsheet

Start with one representative interaction. A client sends a request, an edge service authenticates it, application code retrieves context, a router selects an inference destination, and the model returns an answer. The application may validate that answer, call another service, and stream the result. Write down every separately metered boundary along this journey.

Use the edge network architecture guide to distinguish delivery work from origin work. Then map AI routing decisions to the model calls they actually create. A single visible response might involve a classification call, a generation call, and a repair call. Counting only the last call produces a misleading estimate.

For each boundary, record its usage unit, assumed quantity, rate, owner, and measurement source. Keep a separate row for shared costs. The first useful result is a cost inventory that engineers can reconcile with running software.

Include work outside the synchronous response, such as document ingestion, embedding refreshes, scheduled evaluations, and model release distribution. Allocate those activities to the feature or customer population they support. If they serve several features, state the allocation rule and retain the unallocated total for reconciliation.

2. Define both accounting and product units

Choose a consistent cost basis

The FOCUS billing specification provides common terminology and a schema for comparing cost data. Its billed cost and effective cost concepts serve different purposes: billed cost supports invoice views, while effective cost accounts for applicable discounts and allocated prepaid purchases. This distinction matters when a capacity commitment is paid in one period but supports work across many periods.

Choose the basis appropriate to the question, label it, and keep it consistent across numerator and reporting window. Do not add a prepaid purchase to the usage costs that already include its allocation.

Choose a useful outcome

Track cost per submitted task and cost per successful task. Define success explicitly: for example, a valid answer delivered inside the latency target and accepted by an evaluation rule. Also retain input tokens, output tokens, delivered bytes, and inference attempts. These explain movement in the business unit without pretending that every request requires equal work.

3. Build an illustrative monthly scenario

The following values are hypothetical engineering assumptions, not vendor prices or a forecast. Imagine one million submitted tasks per month. An authorized response cache handles 20 percent, leaving 800,000 initial model calls. Internal retries add two percent of those calls, or 16,000 attempts. There are therefore 816,000 billed model attempts, each using 1,000 input tokens and 250 output tokens.

The total is $1,922.50, or $1.9225 per thousand submitted tasks. If 960,000 tasks meet the success definition, the cost becomes approximately $2.00 per thousand successful tasks. Internal inference retries do not create additional incoming client requests in this particular scenario. Transfer and telemetry quantities already include the assumed retry traffic.

4. Test assumptions that multiply work

Averages conceal expensive combinations. Long prompts may occur on the same requests that require a larger model, generate longer answers, and trigger retries. Capture representative workload groups instead of multiplying unrelated global averages together.

In the example, increasing the retry rate from two to ten percent raises model attempts from 816,000 to 880,000. At the assumed token lengths and rates, the additional 64,000 attempts cost $128. The modeled total becomes $2,050.50 if every other row stays constant. In a real estimate, recalculate the accompanying transfer, compute, and telemetry as well.

Similarly, reducing the response cache hit rate from 20 to ten percent creates 918,000 attempts at the original retry rate. Those additional 102,000 attempts add $204 in model cost alone. State whether a retry percentage means extra attempts per initial call or the share of calls that retry; multiple attempts make those different quantities.

5. Include capacity, delivery, and observation

Managed model charges and owned inference capacity need different calculations. For owned capacity, budget provisioned time, useful utilization, reserved headroom, and idle periods. Divide the complete capacity cost by completed work. A busy accelerator during a short load test does not establish monthly utilization.

Model delivery introduces its own traffic: initial downloads, updates, rollback artifacts, and transfers between storage and serving locations. Specify which boundaries incur charges. Separate bytes delivered to users from bytes moved internally, and avoid charging the same transfer twice merely because two systems recorded it.

Observability also needs a quantity model. Estimate event size, event frequency, retention, query activity, and any indexing charges separately. Reducing verbose payload logging can lower both exposure and expense, but preserve enough signals to explain failures. Record the measurement coverage beside the cost number so that an apparent saving cannot hide lost visibility.

6. Evaluate savings against service quality

Several changes can reduce inference demand: shorter context, bounded outputs, reusable preprocessing, safe response caching, and routing simple tasks to smaller models. Each changes behavior, so compare it against a stable set of tasks with explicit quality and latency requirements.

The guide to LLM request routing explains why route selection should consider more than a nominal model rate. A cheaper first attempt can become expensive if escalation is frequent. Budget the entire routing tree, including classification and fallback calls, then measure how often each branch runs.

Before assigning savings to a cache, review the rules for caching API responses. An answer may be reusable only within a tenant, permission set, model release, and freshness window. Estimate the hit rate for that valid key space. A spreadsheet assumption that depends on sharing private responses is an architectural defect.

7. Connect forecasts to operational ownership

Give each cost component an owner and a measurable driver. Product teams can forecast task volume and workload mix. Engineers can supply token distributions, attempt counts, utilization, and traffic quantities. Finance or procurement can confirm the applicable contract rates and accounting treatment. Keep the assumptions together so that a change has a visible explanation.

Maintain baseline, growth, and stress scenarios. The stress case should include a plausible failure, such as reduced cache reuse during a release or increased fallback during an endpoint outage. Separate usage uncertainty from pricing uncertainty; they need different responses.

Forecast by cohort when adoption changes the workload. A new customer importing a document archive may consume much more ingestion capacity than an established customer asking routine questions. Showing that temporary cost separately prevents a launch month from becoming an inaccurate estimate of steady operation.

Review actuals against estimates at a regular cadence. Attribute variance to volume, mix, efficiency, rates, or shared allocations. Set operational limits on runaway work, including retry budgets and maximum output lengths. A budget alert is most useful when its owner has a specific control to adjust.

Conclusion: make the budget a living engineering model

A useful AI infrastructure budget is a small, inspectable model of system behavior. It connects requests to attempts, attempts to resources, and resources to costs. It separates assumptions from measurements and submitted work from successful outcomes.

Start with the largest uncertain drivers, keep the arithmetic reproducible, and refine the estimate as production evidence arrives. When someone proposes a routing, caching, or deployment change, use the same model to ask what work disappears, what work moves, and what new expense appears. That makes cost a practical input to architecture and service quality.