Small models fail differently than large ones, and the evaluation practices the field inherited from large-model leaderboards are close to useless for deciding whether a fine-tuned 1B–8B model is safe to put behind a production route. A frontier model that regresses on a benchmark usually degrades gracefully; a small model that regresses often falls off a cliff — it stops emitting parseable structure, collapses into repetition, or silently loses a capability that a single aggregate score never measured. If your evaluation apparatus reports one number, was benchmarked on data the model plausibly saw, and runs through the same serving stack as the model under test, you do not have an evaluation. You have a ritual.
We run a small-model program inside a production AI operating system — Model Forge, our dataset-capture → fine-tuning → serving → evaluation loop — and the evaluation layer is the part we have rebuilt most often. This page describes what survived: multi-metric judge panels anchored on deterministic scoring, benchmark hygiene treated as a supply-chain problem, contamination testing that models the realistic leak paths rather than the theatrical ones, and a fail-closed gate that is architecturally forbidden from sharing a route, a dependency, or a failure domain with the model it judges.
Why single-score evaluation lies about small models
The central statistical fact is that fine-tuning a small model is a zero-sum reallocation of limited capacity. Improve instruction-following on your target distribution and you will, with high probability, degrade something else — formatting discipline, refusal behavior, long-context recall, or a capability nobody thought to measure. Aggregate scores are built to hide exactly this. A model can gain four points of average while losing a capability your production traffic depends on, because averages launder regressions through improvements.
Large-model evaluation partially escapes this because capacity slack absorbs the reallocation. Small-model evaluation cannot. So the first design commitment is that an evaluation result is a vector, never a scalar, and gating decisions are made per-dimension against per-dimension thresholds. The moment you collapse the vector to rank checkpoints, you have re-created the problem. We rank nothing; we gate everything.
The second fact is that small models fail structurally before they fail semantically. Malformed tool calls, broken schemas, truncated outputs, and degenerate repetition are the leading edge of regression, and they are detectable deterministically — no judge model required. This ordering matters: the cheapest, most reliable detectors should fire first, and expensive, fallible detectors should only ever refine what the cheap ones admit.
The judge panel: deterministic floor, model-judge ceiling
Our panels are layered, and the layering encodes an epistemic hierarchy: each layer is only allowed to be as generous as the layer below it permits.
Layer 1 — deterministic, versioned scorers. Schema validity, exact-match and normalized-match correctness where ground truth exists, structural constraints (did the model emit the required fields, in bounds, in order), latency and length envelopes, repetition and degeneration detectors. These scorers are versioned artifacts in their own right: a scorer change is a change to the meaning of every historical score, so scorer versions are pinned per evaluation run and a scorer upgrade triggers re-scoring of the reference set, never silent reinterpretation. This is the same discipline we apply in our deterministic scoring engines elsewhere in the platform: the scoring function is code, code has versions, and comparing scores across scorer versions is a category error.
Layer 2 — evidence-gated model judges. LLM judges are useful and untrustworthy in equal measure, so we constrain them the way one constrains an unreliable witness: they may only assert what they can cite. A judge verdict that does not quote or point to the specific span of the model output justifying the verdict is discarded as unsupported — the judge saying "this answer is wrong" is worth nothing; the judge saying "this answer is wrong because this span contradicts this reference fact" is checkable, and checkable is the whole point. Evidence-gating converts judge output from opinion into an auditable claim, and a meaningful fraction of judge verdicts fail the evidence check. Those failures are signal too: a rising unsupported-verdict rate is an early indicator of judge drift.
Layer 3 — panel disagreement as a first-class metric. Multiple judges (different models, different rubric framings) score the same outputs, and we track agreement explicitly. High disagreement on a slice does not average away into a middling score; it flags the slice as unevaluable as configured, which routes to a human or to rubric repair. Known judge pathologies — position bias, verbosity preference, self-preference toward outputs stylistically similar to the judge's own family — are the standard reasons single-judge setups mislead, and panel disagreement is the cheapest instrument that surfaces them.
The hierarchy has a simple invariant: a model judge can lower a verdict below the deterministic floor's ceiling, but can never raise a verdict above it. If the output failed schema validation, no amount of judge enthusiasm rescues it. This sounds obvious written down; in practice most judge-based pipelines we have reviewed allow exactly this inversion, usually by summing weighted scores across layers.
Benchmark hygiene as supply-chain discipline
We treat evaluation sets the way a serious build system treats dependencies: pinned, hashed, provenance-tracked, and rotated deliberately.
- Frozen and fingerprinted. Every evaluation set version is content-addressed. A gate result names the exact set version it ran against, and results across set versions are never compared as if commensurable.
- Provenance recorded at capture time. Because our evaluation items are substantially derived from captured production traffic (that is what Model Forge's capture stage is for), every item carries its origin. This is what makes contamination auditing possible at all — you cannot audit leakage of data whose lineage you never recorded.
- A quarantined split. A portion of every set is held out from everyone: not in fine-tuning data, not in prompt-engineering loops, not visible to the engineers iterating on the model. It exists to answer one question — did we overfit to our own evaluation? — and it is rotated once used, because a quarantine split loses its meaning the first time a decision is made against it.
- Blueprinted like any other change. Under our Codex discipline, an evaluation-set change ships with an executable blueprint and runbook like any other production change, and the dependency graph at ticket.lucaexpress.com lets us answer "which gate decisions depended on set version X" mechanically rather than archaeologically. Evaluation sets are load-bearing infrastructure; we version them like it.
Contamination realism
Most contamination discussion fixates on the dramatic case — the benchmark was in the pretraining corpus — which, for teams fine-tuning small models on their own captured data, is usually the least likely leak path. The realistic paths are mundane and internal:
- Capture-loop leakage. The same production traffic that seeds fine-tuning data also seeds evaluation data. Without lineage-based splitting, near-duplicates of training items land in the eval set and the model grades itself on its own homework. This is the dominant contamination mode in capture-driven pipelines and it is entirely self-inflicted.
- Iteration leakage. Engineers eyeball eval failures, adjust data or prompts, re-run. No item ever entered a training set, yet after enough cycles the team has overfit to the benchmark on the model's behalf. The quarantined split exists precisely for this.
- Judge-side leakage. The judge model has seen the benchmark, or the rubric, or canonical answers to it — inflating agreement with reference answers and deflating credit for valid novel ones. Judge contamination is almost never audited, and panels with heterogeneous judge lineages are the only practical mitigation we know of.
Honest contamination work also means honest confessions: n-gram overlap detection misses paraphrase-level leakage, embedding-similarity screens have tunable thresholds that can be tuned into meaninglessness, and no screen proves absence of contamination. The defensible claim is never "this benchmark is clean"; it is "these specific leak paths were audited by these specific mechanisms, on this lineage record."
The gate must not share a route with the model it judges
The principle we hold most firmly is architectural, not statistical: the evaluation gate is a control plane, the model under test is a data plane, and they must share nothing — not a serving route, not a runtime dependency, not a deployment unit, not a failure domain.
candidate model evaluation gate
┌────────────────┐ outputs only ┌──────────────────────┐
│ serving stack │ ───────────────► │ deterministic floor │
│ (data plane) │ │ evidence-gated panel │
└────────────────┘ │ threshold check │
▲ └──────────┬───────────┘
│ promote / hold / roll back │
└─────────────── control ◄────────────────┘
(gate degraded or unreachable ⇒ HOLD, never pass)The reasons stack:
- Shared failure domains convert outages into approvals. If gate and model share infrastructure, the failure that degrades the model can simultaneously degrade the gate's ability to notice — the exact correlated failure the gate exists to catch.
- Fail-open is the default unless you engineer it away. In any pipeline under delivery pressure, an unreachable evaluation step gets skipped, retried into a timeout that "passes," or waived. Fail-closed must be structural: the gate's absence of an explicit pass verdict blocks promotion. Silence means no. We learned this inside a multi-agent continuous-delivery pipeline, where several AI coding agents ship one codebase through a serialized merge queue and release trains — agents optimize for shipping, and any gate that can be routed around by an agent under throughput pressure eventually will be. Gates survive there only if bypassing them is not an expressible operation.
- Judge and judged must not converge. If the judge is served by the same stack, upgraded by the same process, or fine-tuned from the same lineage as the candidate, evaluation trends toward self-assessment. Small-model programs are especially exposed because the temptation to use your own cheap model as its own judge is strong and the self-preference literature says it is exactly wrong.
- Auditability requires independence. A gate verdict is a record — model version, eval-set hash, scorer versions, judge panel composition, per-dimension results — that must be reconstructible after the fact. In our system that record lands in the Codex alongside the change it gated, which is only meaningful if the gate's account of events cannot have been altered by the thing it judged.
Trade-offs and failure modes, candidly
This apparatus is not free, and it has its own pathologies. Fail-closed gates convert evaluation-infrastructure flakiness directly into delivery latency; when the gate is the unreliable component, every gate outage is a blocked train, and the organizational pressure to add a bypass grows with each incident — the correct response is investing in gate reliability, and the tempting response is the waiver flag that quietly ends fail-closed. Per-dimension thresholds proliferate and each threshold is a judgment call that can be argued; threshold governance is unglamorous ongoing work. Evidence-gated judging raises cost per verdict substantially over naive judging, and quarantine-split rotation means periodically paying to rebuild trust you previously had. Panel disagreement flags real ambiguity but also flags badly written rubrics, and distinguishing the two takes human time.
The deepest limitation is that all of this measures the distribution you thought to encode. A gate is a hypothesis about what matters; production traffic falsifies hypotheses on its own schedule. We treat gate escapes — regressions that passed the gate and surfaced in production — as the primary input for evaluation-set evolution, which closes the loop but never finishes it.
Where this is heading
Three directions occupy us. First, tightening the capture-to-eval lineage so contamination auditing becomes a static property of the pipeline rather than a periodic investigation. Second, judge panels that are themselves gated — small, purpose-trained judge models whose own promotion passes through the same fail-closed machinery, with the route-separation invariant enforced recursively. Third, richer disagreement analytics: treating panel disagreement structure, not just rate, as a diagnostic for which capability dimensions our rubrics fail to specify. Elements of the scoring and gating machinery described here are part of our patent-pending portfolio.
We take on external research engagements — evaluation architecture, small-model programs, and fine-tuning pipelines included. If your gate can be talked past, we should talk first.