Every system that calls a language model makes a decision that is usually invisible: which model. In most codebases that decision is a string literal — a model identifier hardcoded at the call site, copied from the last call site, updated by grep when the provider deprecates something. This is tolerable when one team ships one product against one model. It stops being tolerable the moment you operate what we operate: an AI operating system serving dozens of distinct principals — human users on different license tiers, cognitive personas with different roles, autonomous coding agents, background evaluation jobs — across a heterogeneous fleet that mixes frontier models with small fine-tuned specialists produced by our own Model Forge pipeline.
At that point, "which model" is no longer an engineering detail. It is a policy decision with cost, capability, licensing, and safety consequences, and it deserves the same treatment we give any other policy decision in an operating system: centralized resolution, explicit ceilings, audited changes, and a clear story for what happens when the preferred answer is unavailable.
This page describes how we restructured model selection in Luca AI Express from hardcoded assignment to policy resolution, and the four mechanisms that made it survivable in production: tier ceilings, lattice-respecting fallbacks, drift guards, and fail-closed judge routes. Aspects of this work are patent-pending; we describe the architecture and reasoning here, not the implementation.
Model selection is an authorization problem
The reframe that unlocked the design: model selection is structurally identical to authorization. In classical access control, code never asks "am I root?" — it asks "may this principal perform this action on this resource?", and a policy engine answers from role and grant data. We apply the same shape to models. Call sites never name a model. They name a task class — a semantic description of what they need ("conversational reply for a persona," "structured extraction," "scoring judgment," "bulk summarization") — and identify the principal on whose behalf the work runs. A resolver maps (principal, task class, context) to a concrete model assignment at request time.
Three things fall out of this immediately.
First, license becomes enforceable. A customer's plan can entitle them to a maximum model tier, and that ceiling is enforced in one place rather than promised in a spreadsheet and hoped for in code. Second, model migration becomes a policy edit, not a codebase-wide refactor. When a provider deprecates a model, or when we promote a fine-tuned specialist into a route, we change a mapping, not four hundred call sites. Third, delegation gets a defined semantics. When an autonomous agent acts on behalf of a user — pervasive in our multi-agent delivery pipeline — the effective ceiling is the meet of the agent's ceiling and the user's ceiling. Nobody escalates model access by routing a request through a more privileged intermediary. This is the same confused-deputy reasoning that motivates capability systems, applied to inference.
The resolution pipeline
Conceptually the resolver runs a short, deterministic pipeline:
request(principal, task-class, context)
│
▼
┌──────────────┐ role/license ┌──────────────┐
│ identify │─────────────────▶│ compute │ ceiling = meet of all
│ principal(s) │ (incl. on- │ tier ceiling │ applicable ceilings
└──────────────┘ behalf-of) └──────┬───────┘
▼
┌──────────────┐
│ resolve task │ task-class → ordered
│ class route │ candidate chain
└──────┬───────┘
▼
┌──────────────┐
│ clamp chain │ drop candidates above
│ to ceiling │ ceiling; check floors
└──────┬───────┘
▼
first available candidate
(or fail closed, per route)Each task class carries an ordered candidate chain — preferred model first, then acceptable degradations. The ceiling clamps the chain from above; some routes also carry a floor, a minimum tier below which the route must not execute at all. The gap between "degrade" and "refuse" turns out to be the most safety-relevant line in the whole design, and we return to it below.
Resolution is deterministic and versioned: the same request against the same policy version yields the same assignment. That determinism is not pedantry — it is what makes incidents debuggable and evaluations reproducible. When behavior changes, we want to be able to say whether the model changed or the policy changed, and prove it from logs.
Ceilings are ordered; capability is not
An honest caveat about the tier lattice: ordering models by "capability" is a modeling fiction. A small model fine-tuned on ten thousand examples of one domain task routinely beats a frontier model on that task while being strictly worse everywhere else. A single total order cannot express that.
Our resolution: the lattice orders cost-and-generality tiers, not task quality. Specialist models do not live "below" frontier models on a quality axis; they live on routes. A Model Forge specialist gets attached to specific task classes where its evaluation record justifies it, and the tier system merely governs what a principal is entitled to spend. This separation matters commercially too: it lets us serve a fine-tuned small model to a low-tier license on a route where it genuinely outperforms, without pretending it is a general substitute for a frontier model. The ceiling answers "how much model may this principal consume"; the route answers "which model is actually right for this job."
This is also, in our view, the missing deployment story for small LLMs generally. Most of the literature on distillation and fine-tuning stops at the eval table. The operational question — how does a specialist safely enter a production traffic path, and how does it get demoted when it regresses — is a routing-policy question. Treating routes as governed, versioned objects gives specialists a promotion path: introduce the model as a shadow candidate, compare against the incumbent on live traffic, promote it into the chain by policy edit, and demote it the same way. Every one of those promotions ships through our Codex discipline — an executable blueprint, playbook, and runbook accompany the policy change — so the reasoning behind a route assignment survives the engineer who made it.
Fallbacks that respect the lattice
Fallback chains exist because models fail: quota exhaustion, provider incidents, regional degradation. The naive design — "on error, try the next model" — has two failure modes we consider disqualifying.
The first is falling up: a fallback that lands on a model above the principal's ceiling, silently converting an outage into a licensing violation and a cost incident. The clamp must apply to the entire chain, not just the head. The second is silent falling down: a route quietly serving a weaker model for hours because nobody distinguishes "degraded" from "healthy." Degradation must be a first-class, observable state — logged with the resolved assignment, surfaced in telemetry, and bounded in duration by alerting, so that a fallback is a bridge and never a new normal.
There is also a fleet-dynamics hazard: when a popular tier fails, every route falls back to the same secondary at once, and the failover itself becomes a stampede. Chains need to be designed fleet-wide, with an eye on aggregate capacity of each fallback target — another reason this belongs in one policy layer rather than scattered across services. The dependency graphs we maintain at ticket.lucaexpress.com earn their keep here: because routes, task classes, and consuming modules are nodes in a browsable graph, we can answer "what co-fails if this tier goes down" before the incident instead of during it.
Fail-closed judge routes
The sharpest design decision in the system is an asymmetry. Most routes fail open-ish: a chat persona degrading to a smaller model is a worse experience, but a defensible one. Judge routes fail closed.
Judges, in our architecture, are the models embedded in evidence-gated scoring passes — deterministic, versioned scoring engines where an AI pass renders a judgment that gates something: a quality score, a release-train check, an evaluation verdict that decides whether a Model Forge candidate gets promoted. Substituting a weaker judge does not degrade the experience; it corrupts the measurement. A gate that quietly swaps its judge during an outage will happily pass work the real judge would have failed, and you will discover this weeks later, in the worst possible way: as unexplained downstream quality drift with a clean audit trail.
So judge routes carry a floor equal to their preferred tier and an empty effective fallback chain. If the designated judge is unavailable, the judgment does not happen — the work queues, the gate holds, and humans get paged. We accept availability loss on evaluation paths to preserve the integrity of everything those evaluations gate. There is a subtler corollary: judge assignments are pinned to exact model versions, never floating aliases. A provider silently rotating an alias underneath a judge is the same corruption in slow motion. Reproducible measurement requires that the measuring instrument be immutable between calibrations.
Drift guards
Indirection decays. The failure mode of every policy layer is that engineers — and, in our shop, AI coding agents, which are diligent pattern-matchers of whatever the codebase already contains — reintroduce hardcoded model identifiers at call sites, and the policy layer becomes a decorative facade over a shadow routing system. We treat this as a continuously-defended invariant, guarded at three levels:
- Statically: automated checks in the merge queue reject literal model identifiers outside the policy layer. With multiple autonomous agents shipping into one serialized queue, this class of gate does most of the work; agents learn the constraint quickly because the queue enforces it uniformly.
- At runtime: the resolved assignment travels with the request, and the serving edge verifies that what executes matches what was resolved. Divergence is an incident, not a log line.
- In governance: routing policy versions are Codex entries like any other change — blueprint, playbook, runbook — so every route's history, rationale, and rollback procedure is queryable rather than tribal.
The runtime check matters more than it first appears. Static gates catch the honest mistakes; the runtime comparison catches the interesting ones — caching layers replaying stale assignments, retry wrappers with their own opinions, SDK defaults asserting themselves when a parameter goes missing.
Trade-offs, candidly
The design has real costs. Debuggability tax: "which model produced this output" now requires consulting resolution logs instead of reading the call site; we consider the versioned logs a net improvement, but the first week of operating this way is disorienting. Policy sprawl: task classes multiply, and an over-grown taxonomy is hardcoding with extra steps — we prune deliberately and treat a new task class as a design decision requiring justification. Latency: resolution adds a hop; ours is a cheap deterministic lookup, but anyone contemplating a learned router in this position should price the inference cost of routing against the inference being routed. Eval coupling: any output cache or stored evaluation is implicitly keyed by model version, so a policy edit must invalidate downstream artifacts — we learned this by being burned, and route-change playbooks now enumerate the blast radius. And fail-closed is a genuine availability cost: gates hold during judge outages. We keep that trade because the alternative — measurements you cannot trust — is a debt with compounding interest.
Where this is heading
Three directions. First, evidence-driven promotion: closing the loop so a Model Forge specialist's live comparison record can propose (never enact) its own route promotion, with humans and Codex governance in the loop. Second, richer principal semantics: ceilings that account for context — the same agent may warrant different tiers in exploratory versus release-gating work. Third, cross-fleet capacity-aware chains: fallback design that reasons about aggregate load at failover time rather than per-route preference alone. The through-line is unchanged: model selection is policy, policy is versioned, and anything that gates must fail closed.
Gus IT Research Institution takes on external research and engineering engagements in this area — model fleet governance, small-model deployment paths, and evaluation-gated delivery. Reach us through isalabresearch.com.