If your app calls more than one LLM provider, something has to decide which one gets each request. Most teams start with a fixed fallback list: try the main model, and if it fails, try the next. That works until traffic grows, and then the same first target gets every call, even when it is slow today, costs ten times what the call needs, or keeps returning answers you throw away.

LLM routing fixes that by choosing the order of targets per call, not once for the whole app. This post covers what routing can do, with examples from vern-llm-router, an open source TypeScript middleware for vern-llm.

What LLM routing means

Routing is deciding, for each request, which provider and model to try first and which to keep as backups. A good router uses four kinds of input:

  • What the call is: who sent it, whether it uses tools, how large it is
  • What each target costs
  • How each target has been performing: latency and answer quality
  • Whether each target is healthy right now

A router that only reorders targets is also safer to adopt. vern-llm-router never adds a target, never changes what a call returns or throws, and if deciding fails the call runs exactly as it would without it.

Route by what the call is

The simplest win is rules. Free tier traffic does not need your best model, and calls with tools should never reach a model that cannot use them.

const router = routerMiddleware({ health: () => llm, routes: [ { name: 'bulk', when: { context: { tier: ['free', 'trial'] } }, targets: ['small'] }, { name: 'tools', when: { hasTools: true }, exclude: ['small'] }, { name: 'default', targets: [['main'], '*'] }, ], capabilities: { small: { contextWindow: 32_000, tools: false } }, }); 

The first route that matches decides the order. Routes can match on context values, tools, JSON output or input size, and capabilities skips a target only when a request is known not to fit it.

Rank providers by effective cost

Price per call is the number everyone compares, and it is misleading. A cheap model that returns unusable answers 60% of the time costs more per usable answer than a pricier one that rarely fails.

The router ranks on effectiveCost: the price divided by the share of usable answers, measured from your own traffic.

const router = routerMiddleware({ stats: logger.stats, // from vern-llm-logger cost: usage.costFor, // from vern-llm-usage routes: [{ name: 'default', strategy: { type: 'rank', weights: { effectiveCost: 1 } } }], }); 

In a run with three providers priced at $0.002, $0.004 and $0.006 per call, and 60%, 2% and 5% of answers rejected, ranking by plain cost puts the cheapest provider first. Ranking by effective cost moves the $0.004 provider to the top, because the cheapest one wastes most of its spend.

Ranking can also use latency, predicted latency (time to first token plus typical answer length), and quality. Scores are on a log scale, so units combine and one outlier only hurts itself. A small share of calls tries a random usable target, so no provider stops being measured.

Keep prompt caches warm with sticky routing

Provider prompt caches work per provider. If one user's calls are spread across providers at random, every call starts cold.

A weighted strategy with stickyBy hashes a context value instead of drawing at random, so a user keeps the same first target:

strategy: { type: 'weighted', weights: { primary: 8, canary: 1 }, stickyBy: 'userId' } 

The same weights also give you a canary: send one call in nine to a new model, and keep each user on the same side.

Move unhealthy providers back, then ease them in

A router should know what your circuit breakers and rate limits already know:

  • A target with an open breaker, or an exhausted rate limit, moves behind the healthy ones. It is never dropped, so it is still used if nothing else is left.
  • A provider 429 starts a cooldown for the Retry-After it sent, 5 seconds without one and at most 60.
  • A target whose breaker just closed ramps back from 10% of traffic to full over 30 seconds by default, so a recovering provider is not hit with everything at once.
  • Soft limits on latency, cost or quality failure rate move a target behind the others instead of removing it.

Roll out routing without risk

Changing routing in production is the scary part, so the router gives you ways to check first:

  • router.preview() returns the decision a call would get right now, without calling anything.
  • apply: false is shadow mode. Every decision is computed and emitted as a router.decision event, and no call changes.
  • The driftDimension logger dimension flags calls where the router's first pick is not the target that answered, so you can judge a rule before switching it on.

Try it

npm i vern-llm-router 

vern-llm is a peer dependency and the router has no other dependency. Docs: vernllm.dev/docs/extensions/router. Source: github.com/LakBud/vernLLM.

What routing rule has been the most useful in your own LLM setup? Tell me in the comments.