AI Cost Is an Architecture Problem

AI teams often discover cost in the same way they discover an unbounded loop: through an alert after the system has already behaved exactly as designed.

The model is blamed first. Prices are compared. Prompts are shortened. A smaller model replaces a larger one in a few obvious places.

Those changes can help. They miss the deeper cause.

An AI system spends money through architecture: how often it invokes intelligence, how much context it assembles, which failures trigger retries, how work is decomposed, when humans enter, and what evidence is required before completion. The price per token is only the exchange rate applied to those decisions.

Cost is not a tax on intelligence. It is a property of the workflow.

Start With Cost Per Completed Outcome

Cost per request is easy to measure and easy to misuse.

One request may classify a message. Another may run for an hour, edit several files, and verify a deployment. Comparing their token totals says little about efficiency. A cheap request that fails and sends work to a human may cost more than an expensive request that resolves the case.

The useful denominator is a completed, acceptable outcome: a support issue resolved without reopening, a migration merged with passing tests, a report approved without correction.

Track inference, retrieval, tool, verification, and human-review costs against that outcome. Include retries and abandoned branches. Then pair cost with quality and consequence. The cheapest route that frequently produces rework is not efficient; it has externalized its bill.

Context Has a Carrying Cost

Large context windows create the temptation to send everything.

Every extra document costs tokens on the current call, but the larger expense may be behavioral. Irrelevant context can obscure authority, increase latency, and lead the model down an unnecessary branch. That branch calls more tools, produces more text, and demands more verification.

Treat context as a working set with an admission policy. Include the instructions, evidence, and state necessary for the current decision. Keep the rest addressable through retrieval and artifacts.

A context manifest should make cost visible by source. Teams can then see whether a bulky runbook, duplicated transcript, or verbose tool schema is being paid for on every step. Context engineering becomes cost engineering when the same information boundary is multiplied across thousands of invocations.

Compression is useful only when it preserves what the next step needs. A cheap summary that drops a constraint can create an expensive failure later.

Spend Capability Where Uncertainty Lives

Sending every step to a frontier model is wasteful. Sending every step to a small model and escalating only after failure can be just as wasteful.

The workflow should locate uncertainty.

Deterministic parsing, known transformations, status checks, and policy lookups often do not need a model at all. Classification and formatting may need a fast model. Ambiguous planning, conflicting evidence, and difficult synthesis deserve stronger reasoning. Consequential verification may need a separate model or non-model test.

This is the practical value of model routing: not a race to the cheapest endpoint, but deliberate allocation of capability.

Use a strong model early when it can prevent a long wrong branch. Use a small model late when the answer has already been constrained by verified artifacts. Cost-aware architecture spends intelligence where it changes the decision.

Loops Need Economic Stopping Conditions

Agent loops can turn uncertainty into activity.

The agent searches again, asks another subagent, rereads the same file, or rewrites an answer that already meets the requirement. Each action appears locally reasonable. Together they produce cost without new evidence.

A loop should know what progress means. Record which uncertainty the next action is expected to reduce, the evidence it should produce, and the budget available. If repeated actions do not change the state, the workflow should stop, change strategy, or escalate.

Budgets can be expressed in tokens and dollars, but also in tool calls, elapsed time, retrieval rounds, or model escalations. They should vary by outcome value and risk. A security investigation earns more exploration than a routine tag suggestion.

Budget exhaustion is not success or failure. It is a decision point where the system must return its evidence, uncertainty, and recommended next step honestly.

Parallelism Trades Time for Duplication

Subagents can reduce wall-clock time while increasing total work.

Three researchers may independently rediscover the same sources. Two implementers may produce incompatible approaches that require a costly integration pass. A reviewer may receive several verbose artifacts and spend more tokens reconciling them than one worker would have spent doing the task.

Parallelism is economical when branches are truly independent, the handoff is small, and the cost of waiting is greater than the cost of duplication. It is expensive theater when the decomposition exists only to make the workflow look agentic.

Estimate the merge cost before spawning work. Give each branch a distinct question, a bounded context, and an artifact contract. The subagent decision should be a resource-allocation decision, not a default habit.

Retries Are Part of the Bill

Tool failures, rate limits, malformed outputs, and lost acknowledgements all create repeated work.

Blind retry policies can pay for the same reasoning and context several times. Worse, an effectful retry can duplicate the business consequence and create remediation cost outside the AI platform.

Classify failure states. Retry only when another attempt is likely to help. Resume from a durable checkpoint instead of rebuilding the task. Reconcile ambiguous effects before invoking the model again. Cache stable tool results where policy permits.

Reliability work often pays for itself because it removes entire repeated branches, not just a few tokens from a prompt.

Verification Should Match Consequence

Verification costs money. Skipping it can cost much more.

Not every output needs a second frontier model and an exhaustive test suite. A low-risk draft may need only schema validation. A code change may need targeted tests and static checks. A financial action may require an independent ledger read and a policy record.

Define assurance tiers. Each tier specifies the evidence required, the preferred verifier, and the maximum cost relative to the operation. This prevents teams from applying expensive review everywhere or none at all.

The key is to count verification as part of delivery rather than as overhead. An unverified answer is a cheaper artifact and a different product.

Cache Evidence, Not Conclusions

Caching a model response can reduce cost and quietly preserve a stale answer.

The safer unit to reuse is often the expensive evidence: parsed documents, embeddings, normalized tool results, test artifacts, or a verified intermediate representation. A later step can reason over those artifacts under the current policy.

When conclusions are cached, bind them to source versions, model and harness versions, applicable policy, and an expiration condition. Invalidate them when any dependency that affects meaning changes.

The goal is not maximum cache hit rate. It is avoiding the same expensive work while remaining honest about freshness.

Give Cost an Owner and a Trace

A monthly total cannot tell a team what to improve.

Trace cost through the same hierarchy used for observability: product outcome, workflow, step, model invocation, context source, tool, and retry. Record the route and verification tier. Attribute background agents and evaluation traffic separately from user-facing operations.

Then review expensive outcomes, not only expensive calls. A single large invocation may be efficient. A swarm of tiny calls may be the real leak.

Cost regressions should enter the release pipeline beside quality regressions. If a new prompt improves success by one percent and doubles inference, the team should make that trade consciously.

Build for Useful Intelligence, Not Cheap Tokens

Cost discipline is sometimes framed as using less AI. The better goal is using AI where it has leverage.

Measure completed outcomes. Keep context intentional. Route around uncertainty. Bound loops. Make parallelism earn its merge cost. Design retries and durability. Scale verification with consequence. Trace the result.

The bill is a delayed architectural diagram. It shows how the system decomposed work, where it became confused, and which decisions it repeated.

Design those decisions well, and cost becomes predictable without making the product less capable.