Model Routing Is a Product Policy

The easiest model-routing strategy is to send everything to the most capable model you can afford.

It is also a quiet way to let a vendor leaderboard become your product strategy.

A support reply, a repository migration, a medical-record summary, and a destructive infrastructure change do not need the same reasoning depth, latency, privacy boundary, or tolerance for uncertainty. They should not inherit the same model merely because one endpoint won the last evaluation cycle.

Routing is the point where business intent becomes an execution choice. Done well, it decides how much intelligence to spend, what evidence to require, where data may travel, and when the system should refuse to proceed. That makes routing a product policy, not an optimization trick.

“Best Model” Is Not a Useful Requirement

A model can be best on a benchmark and wrong for the work in front of it.

The strongest reasoning model may be too slow for typeahead suggestions. The cheapest model may be excellent for extracting known fields and unreliable when policy language conflicts. A local model may protect sensitive data but lack the capability to complete a difficult investigation. A frontier cyber model may solve the task and still be prohibited from receiving open network access.

The useful question is not which model is best. It is which execution profile satisfies this task’s contract.

That profile includes more than quality:

• the consequence of a wrong answer
• the value of a fast answer
• the amount and sensitivity of context
• the tools the model may use
• the evidence required before completion
• the cost envelope for the operation
• the fallback behavior when confidence is inadequate

These dimensions belong to the product because users experience their tradeoffs. A fast, shallow answer is a feature in one surface and a failure in another.

Route Work, Not Prompts

Prompt-level routing often begins with a classifier that labels an input as easy or hard. That can help, but the prompt is rarely the whole unit of work.

An agentic task changes as it runs. A cheap model may classify the request, retrieve the obvious files, and produce a plan. The implementation may require a stronger coding model. A consequential tool call may require an independent policy check. A final summary may return to a smaller model once the evidence is fixed.

The route should therefore attach to the current step and its risk, not permanently to the opening message.

This is especially important when a task reveals hidden difficulty. A request that sounds like “rename this setting” may turn into a schema migration across several services. A router should be able to escalate when the evidence changes instead of forcing the original cheap path to bluff its way through.

Routing needs state: what is being decided now, what has already been established, and what consequence follows if this step is wrong.

Define Capability Tiers in Product Language

Teams get into trouble when their routing configuration is a list of vendor model names spread through application code.

Model names change. Product obligations should not.

Define a small set of internal execution profiles such as fast-transform, grounded-analysis, deep-reasoning, sensitive-local, and high-risk-reviewed. Each profile describes required behavior: maximum latency, supported context, tool permissions, data residency, evaluation thresholds, and verification policy.

The router maps a step to a profile. A separate deployment configuration maps the profile to a model and harness version.

That separation lets a team replace a vendor, test a new release, or move sensitive work on-premises without rewriting product logic. It also creates an honest contract. “Use model X” says little about the user experience. “Grounded analysis, no write tools, citations required, regional processing only” says what the system must preserve.

Risk Should Control Both Model and Authority

Routing only the model while leaving every tool available misses the more important half of the decision.

A stronger model does not automatically deserve broader authority. In many cases, the opposite is true: the more capable the model, the more consequential the actions it can discover and attempt.

The route should choose an execution bundle:

• model capability
• context sources
• tool allowlist
• sandbox and network policy
• approval requirements
• verification depth

A low-risk drafting route may use broad reference material and no effectful tools. A deployment route may use a high-capability model inside a narrow repository sandbox, with one approved deployment tool and mandatory post-deploy checks. A sensitive-data route may trade raw capability for local processing and stricter retrieval.

The model is one component of the agent harness. Routing should configure the harness around the task.

Confidence Is Not a Routing Oracle

It is tempting to ask a small model whether it is confident and escalate when it says no.

Self-reported confidence is useful evidence and a weak control. Models can be confidently wrong, especially when context is incomplete. They can also sound uncertain while producing a correct result. A routing policy needs observable signals outside tone.

Useful escalation signals include missing required sources, disagreement among retrieved authorities, failed validation, an expanding change set, repeated tool errors, and a step entering a higher consequence class. The system can also use a lightweight model to generate an answer and a separate evaluator to test it against explicit criteria.

Escalation should answer a concrete failure: the current route cannot satisfy the contract with the evidence available.

Fallbacks Must Preserve Meaning

When a model is unavailable or a budget is exhausted, silently dropping to a weaker model can change the product while leaving the interface untouched.

That is dangerous. A route that promised deep policy analysis should not become fast summarization because the preferred provider timed out.

Fallbacks should be declared per profile. Some tasks can use a second qualified model. Some can degrade to read-only assistance. Some should queue for later. Some must stop and tell the user that the required execution path is unavailable.

The fallback must preserve the contract, not merely return a response.

This is also where multi-provider designs earn their complexity. Redundancy matters only when the alternate route has been evaluated on the same representative work, uses compatible tool contracts, and satisfies the same data policy. A second endpoint is not resilience until its behavior is understood.

Evaluate the Router as a System

A model evaluation asks whether a model can perform a task. A routing evaluation asks whether the system assigns the right resources and controls to that task.

The test set should include easy work that should remain cheap, difficult work that should escalate, sensitive work that must stay within a boundary, and adversarial inputs that try to obtain a more privileged route. Measure final task quality, latency, cost, escalation rate, policy compliance, and the harm from misrouting.

Two mistakes matter differently. Over-routing spends more time and money than necessary. Under-routing produces weak or unsafe outcomes while pretending the task was handled. Optimize them separately.

Production traces should record the route requested, the signals used, the execution profile selected, any escalation, and the final outcome. Without that record, cost changes and quality regressions become impossible to explain.

Make Routing Legible to Users

Not every internal model decision needs to become a settings panel. But users should understand material changes in behavior.

If a task moves into a slower deep-analysis path, say so. If private data requires local processing with reduced capability, expose the tradeoff. If the system cannot satisfy a high-risk contract and needs human review, do not disguise the stop as generic latency.

Legibility builds better expectations. It also prevents the system from using invisible routing to make commitments the selected path cannot keep.

The Route Is Part of the Product

The model market is splitting into fast, deep, open, local, specialized, and tightly controlled capability tiers. That diversity is useful only if the application knows what each kind of work requires.

Describe tasks in product language. Route steps instead of entire conversations. Bind model choice to context, authority, and verification. Define honest fallbacks. Evaluate the router on consequences, not only average cost.

The winning AI product will not be the one that calls the smartest model most often.

It will be the one that makes the right intelligence available under the right conditions—and can explain why.