Model routing as a first-class architectural concern

Four decisions that look like one

Ask a team which model they use and you get a single answer. Ask why, and four separate arguments come out at once: it is cheap enough, it is fast enough, it can run where the data has to stay, and it is good enough at the task. Those are four constraints owned by four different parts of the business, and they change on completely different schedules.

When they are collapsed into one hard-coded choice, any change to one constraint forces a rewrite. A new residency requirement for one jurisdiction should not be a refactor. It should be a routing rule.

Capability is per step, not per system

A single interaction usually contains several jobs with wildly different difficulty. Classifying an incoming request, expanding a query, reasoning over retrieved evidence, drafting a response, checking that draft against policy. Treating all five as one capability requirement means paying frontier prices for four steps that a small model handles indistinguishably.

The prerequisite is per-step evaluation. Without it, downgrading a step is a nervous guess; with it, it is a measured decision you can defend in a design review. This is the single highest-return piece of instrumentation we install on inference-heavy systems.

Latency budgets belong to the interaction, not the model

Teams quote model latency and then discover the interaction is four times slower, because retrieval, tool calls, guardrail checks and a reflection pass all sit in the same wall clock. The useful artefact is a budget for the whole interaction, allocated per step, agreed with whoever owns the user experience.

Once that exists, routing gets obvious options: run the guardrail check concurrently rather than in sequence, stream the first paragraph while the check completes, or send this class of request to a faster model because the accuracy difference does not survive the user leaving. All three are decisions about the budget, not about the model.

Residency is a hard boundary, so make it explicit

Residency does not trade off against anything. If German customer data may not leave the region, no cost or quality argument matters. The failure mode we see is residency enforced by convention — a comment in a config file, an agreement that this service only calls the EU endpoint — rather than by the architecture.

Make it structural. Data classification travels with the request, the router refuses to dispatch to a non-compliant provider, and the refusal is logged as a policy decision rather than an error. Then the answer to “can you prove it never left” is a query, not an assurance.

What the seam looks like in practice

A routing layer worth having is small. It takes the task type, the data classification and the caller’s tier, evaluates rules in a fixed order — residency first, because it is absolute — and returns a provider, a model, a timeout and a fallback. It records the decision alongside the trace. That is roughly it.

What matters is that the rules are configuration rather than code, and that changing them is a governed act with an approver and a version. Model markets move every few months. If a price change or a new regional endpoint requires a release, your architecture is fighting the thing it was supposed to absorb.

Fallbacks are a product decision in disguise

Every router needs an answer for provider failure, and the technical default — retry on the next-best model — is sometimes wrong. Silently downgrading a clinical summarisation step to a weaker model is worse than failing, because nobody downstream knows the standard changed.

So we make the fallback policy part of the workflow definition, with three explicit options: degrade and mark the output, queue and retry, or fail loudly to a human. Which one applies depends on consequence, and that is a conversation for the business owner, not a default in a library.

If you take one thing