What is a model router for AI? A plain-English guide

A model router for AI is a decision layer that picks which large language model answers each incoming request. Instead of your application always calling one model, the router reads the request, applies a policy based on cost, quality, or latency, and sends it to whichever model fits. The choice happens per request, at runtime.

That’s the whole idea. The interesting part is why anyone bothers.

Most teams start with one model because that’s the simplest thing that works. Then the bill arrives and token pricing stops being an abstraction. You look at your traffic and realize that maybe two thirds of it is “summarize this paragraph” and “classify this ticket,” and you’ve been paying frontier-model prices to do work a much smaller model handles fine. Meanwhile the genuinely hard 10% of your traffic, the multi-step reasoning, needs every bit of the expensive model you’re paying for.

A router is how you stop paying the same price for both.

How does a model router work?

The mechanics are less exotic than the marketing suggests. Five things happen:

1. A request arrives at the router instead of going straight to a provider API.
2. The router inspects it. Some routers look only at metadata like which customer sent it or which feature it came from. Others read the prompt itself and estimate how hard it is.
3. A policy runs. This is where “cheapest model that clears the quality bar” or “fastest model available right now” gets decided.
4. The request goes to the chosen model, whether that’s OpenAI, Anthropic, Google, or something you host yourself.
5. The response comes back, along with a log of which model was picked and why.

Step 5 matters more than people expect. If you can’t answer “which model handled this request,” you can’t debug a quality complaint three weeks later.

The routing policy is the product. Everything else is plumbing.

Model router vs AI gateway: what’s the difference?

A model router decides which model handles a request. An AI gateway controls how every request is made, no matter which model gets it.

The confusion is understandable, because most commercial products ship both and the terms get used loosely. Search for either one and you’ll find pages that define them in opposite directions. Here’s the split that actually holds up:

	Model router	AI gateway
Core question	Which model should answer this?	How do we call any model safely?
Decides based on	Prompt content, cost policy, quality targets	Nothing about the prompt
Typical features	Complexity scoring, fallback chains, cost policies	Unified API, key management, rate limits, logging, caching
What breaks without it	You overpay, or you underserve hard prompts	Every service reimplements auth, retries, and logging
Where it sits	Inside the request path, before model selection	Around the request path, wrapping every call

You can run a gateway with no routing at all. Plenty of teams do, and it’s a reasonable first step: unify the API, get logging in place, then decide about routing later. The reverse is rarer, because a router that doesn’t log anything is a router you can’t trust.

What are the main types of AI model routing?

Four approaches show up in production, and they trade control against accuracy in fairly predictable ways.

Rule-based routing

You write the conditions. Requests from the free tier go to the small model. Requests tagged code go to a coding-tuned model. Prompts over 50,000 tokens go to whichever model has the context window for them.

Boring, and boring is the point. You can read the config and know exactly what will happen. AnythingLLM’s model router calls these “calculated rules,” which is a decent name for it: the decision is arithmetic on properties you already have.

LLM-assisted classification

A small, fast model reads the prompt first and labels it. Simple or complex. Creative or factual. Support question or billing question. That label picks the destination.

This handles cases rules can’t express, and it costs you an extra model call on every request. Usually a cheap one, but it’s real latency.

Trained predictive routers

Instead of a classifier bolted on, the router itself is a machine learning model trained to predict which downstream model will perform best on a given prompt. Microsoft’s Foundry model router works this way, and its concept documentation is unusually specific about the tradeoff: in Balanced mode it stays within roughly 1% to 2% of the highest-quality model for a given prompt and picks the cheapest option in that band, while Cost mode widens the band to about 5% to 6%.

Numbers like that are what routing policy should look like. Most vendors won’t give you them.

The academic work behind this category is worth knowing about. RouteLLM, by Isaac Ong and colleagues at UC Berkeley, Anyscale, and Canva, trained routers on human preference data and reported cost reductions of more than 2x on public benchmarks without sacrificing response quality. The routers also generalized to model pairs they weren’t trained on, which is the finding that makes the approach practical rather than a one-off. The paper was published at ICLR 2025.

Semantic and cascade routing

Semantic routing embeds the prompt and matches it against known request types. Cascade routing takes a different bet: send everything to the cheap model first, check whether the answer is good enough, and escalate only when it isn’t. Cascades can save a lot on easy traffic and cost you double on hard traffic.

Approach	Control	Setup effort	Added latency	Best for
Rule-based	Total	Low	Near zero	Known traffic patterns
LLM-assisted	Moderate	Medium	One extra call	Mixed, unpredictable prompts
Trained predictive	Low	Low (if managed)	Small	Teams who want it handled
Cascade	Moderate	High	Variable, sometimes double	Traffic that skews easy

Where the routing decision actually lives

Almost every article on this topic assumes the router is a proxy sitting in your request path. That’s one option out of three, and the other two get ignored.

In a proxy. A separate service receives the call, decides, and forwards it. LiteLLM, OpenRouter, and Portkey all work roughly this way. You get routing plus everything a gateway does, and you accept a network hop and a dependency you didn’t have before.

Inside the model deployment. The router is the thing you call. It ships as a model deployment like any other and fans out internally. Microsoft’s Foundry router is the clearest example, and it means there’s no extra service to run. Its current version, dated 2025-11-18, sits in front of models from OpenAI, Anthropic, DeepSeek, Meta, and xAI behind a single deployment.

In a configuration layer your app reads. This is the pattern nobody writes about. The model choice isn’t made by a proxy at all. It’s a value your application looks up before it makes the call, served by a system built for changing configuration at runtime.

LaunchDarkly’s AgentControl works this way. You replace a hardcoded model name with an SDK call, and the model, prompt, and tool set come back as configuration rather than code. The company describes the change as one where a config update “propagates in under 200ms” and “the deployment pipeline never gets involved.” Because it’s built on the same targeting machinery as feature flags, you can send 10% of traffic to a new model, watch what happens, and roll back without a deploy.

That’s a different shape of routing. There’s no proxy in the request path and no per-request inspection of the prompt. What you get instead is control over model selection by user, segment, or percentage, plus the ability to change your mind in seconds.

It’s the right pattern when your problem is “we can’t switch models without a release” rather than “we’re overpaying on easy prompts.” Those are different problems, and they get solved by different pieces of AI infrastructure. Plenty of teams have the first one and buy a solution to the second.

What are the benefits of model routing in AI systems?

• Lower cost on predictable traffic. The single biggest driver, and the easiest to measure. Move a large share of easy requests down a tier and the bill moves with it, though how far depends entirely on what your traffic mix actually looks like.
• Resilience when a provider degrades. Fallback chains send the request somewhere else instead of returning an error. Microsoft’s documentation is blunt about it: “Failover is enabled by default” and the router “transparently redirects the request to the next most appropriate model.”
• Latency control. If a request needs an answer in under a second, the policy can rule out slow reasoning models before they’re ever called.
• Model portability. When a new model ships, you point some traffic at it and compare. Without a routing layer, that comparison is a code change.
• A higher ceiling on hard prompts. Routing isn’t only about spending less. Sending genuinely difficult requests to a stronger model than your default is a quality gain, and it’s the half of the pitch that gets undersold.

When a model router is the wrong answer

• You’re running one provider and one model. A gateway might help with logging and retries. A router has nothing to decide.
• Your traffic is uniformly hard. If nothing in your workload is safe to downgrade, routing adds a hop and a dependency for no gain.
• You have no evaluation pipeline. Intelligent routing without evals means you’ll discover quality regressions from customer complaints. Build the evals first, then route.
• Latency is your binding constraint. A proxy hop is small but not free. LiteLLM’s published benchmarks put its own overhead at a 12ms median and 29ms at the 95th percentile on a two-instance setup. An LLM-based classifier costs you a full extra model call on top of that.
• You depend on prompt caching. Microsoft states it plainly in its own router documentation: caching benefits apply only when the same model handles consecutive requests with overlapping prefixes. Route aggressively and you can erase savings you already had.
• You think it removes lock-in. It relocates it. You’re no longer coupled to a model provider, and you’re now coupled to the router.

The router also becomes a new single point of failure. Every request in your application now depends on a component that didn’t exist last quarter, and one whose context window ceiling, as Microsoft notes in its own documentation, is set by the smallest model in the pool.

How do you know routing actually worked?

Fix a baseline before you change anything. Record cost per 1,000 requests, p95 latency, and quality on a held-out eval set while everything still runs on one model. Without that snapshot you’ll have opinions instead of results.

Then track four things after the switch:

• Cost per 1,000 requests, split by tier. Aggregate spend hides a router quietly promoting everything to the expensive model.
• Quality on the same eval set. Same prompts, same grader, before and after. A gap here is model drift you introduced deliberately, which means you should be measuring it rather than discovering it.
• p95 latency including router overhead. Measure at your application boundary, not at the provider’s.
• Fallback rate. A rising rate means a provider is degrading and your router is absorbing it. That’s the router working, and it’s also an early warning you’d otherwise miss.

This is a solved measurement problem, academically speaking. RouterBench published an evaluation framework and a dataset of more than 405,000 inference outcomes specifically so routing systems could be compared on something other than vendor claims. Almost nobody uses it in production. Your internal eval set matters more anyway, because it reflects your traffic.

Frequently asked questions

What is an AI router?
An AI router is the same thing as a model router: a layer that chooses which AI model handles each request, based on cost, quality, latency, or rules you define. The terms are used interchangeably, along with “LLM router.”

Is an AI router the same as an AI gateway?
No. A router decides which model gets the request. A gateway handles how requests are made, covering unified APIs, credentials, rate limits, and logging. Most commercial products bundle both, which is why the terms blur.

How do I get started with an AI router?
Log your current traffic for a week and bucket it by difficulty. If a meaningful share is genuinely easy, write one rule sending that bucket to a smaller model and measure quality against a held-out set. One rule, one measurement. Expand only after the numbers hold.

Does model routing hurt output quality?
It can, and whether it does depends entirely on your policy. Microsoft publishes explicit quality bands for its routing modes, roughly 1% to 2% off the best model in Balanced mode. Vendors that won’t state a band are asking you to take quality on faith.

Do you need a model router if you only use one provider?
Usually not for cost reasons, since routing between tiers of one provider’s lineup is something a few lines of application code can do. The case gets stronger if you want failover across regions, or if you want to change models without shipping code.

Where to go from there

The most useful next step isn’t picking a product. It’s spending an afternoon with your own logs, sorting a week of requests by how hard they actually are, and pricing what it would cost to serve the easy ones a tier lower. If that bucket is small, you’ve saved yourself a migration. If it’s large, you now know what routing is worth before anyone pitches you a number.

If you want to see a production routing policy written out in full, Microsoft’s model router documentation is public and specific about how the decision gets made, which is more than most of this category offers.