Mastery Gates: Evaluation Gates a Tuned Model Cannot Argue With

Every team that fine-tunes models eventually ships a bad one. Not because their evaluation harness was missing — almost everyone has one — but because the harness was negotiable. The model scored 71 against a bar of 75, the release was already announced, someone observed that three of the failing items were "arguably mislabeled," the bar moved, and the model shipped. Six weeks later the regression the gate existed to catch showed up in production, and the post-incident review discovered that the gate had been edited four times that quarter, always downward, always with a plausible reason.

We call the fix a mastery gate: an evaluation contract that a tuned model must pass before it is eligible to serve, designed so that the three standard escape hatches — arguing with the gate, backfilling it after seeing results, and weakening it incrementally — are structurally closed rather than merely discouraged. This page describes the design as we run it inside Model Forge, our dataset-capture → fine-tuning → serving → evaluation pipeline, and the reasoning behind each constraint. The mechanism is conceptually simple. Almost all of the engineering is in resisting the social pressure that erodes evaluation systems, and that turns out to be a systems-design problem, not a policy problem.

The failure mode is social, not statistical

The literature on evaluation is mostly about measurement: benchmark contamination, prompt sensitivity, judge bias, statistical power on small eval sets. These are real, and we address them below. But in our experience operating a production AI system, the dominant failure mode of fine-tuning evaluation is organizational. Three patterns recur:

• Argumentation. A model fails; the failing items are re-litigated one by one. Since any individual eval item can be made to look ambiguous under enough scrutiny, a motivated reviewer can always find a reading under which the model "actually" passed. The gate becomes a debate forum.
• Backfilling. The gate is finalized after candidate results are known. Thresholds are set where the current favorite model happens to land. This is the fine-tuning equivalent of HARKing — hypothesizing after results are known — and it converts the gate from a test into a rubber stamp.
• Ratcheting down. Each individual weakening is defensible ("this metric is noisy, widen the tolerance"), and no single change ships a bad model. The sum of ten defensible changes does. Because each edit is small and locally reasonable, drift is invisible without a longitudinal record.

Note what these have in common: none of them can be fixed by a better metric. They can only be fixed by changing who can change what, when, and with what visibility. That is why we treat mastery gates as an immutability and provenance problem first, and a measurement problem second.

What "immutable" has to mean

An immutable gate is not a frozen gate. Evaluation criteria must evolve — tasks change, models expose new failure classes, judges improve. Immutability, as we define it, is a set of four properties borrowed from build systems and pre-registered clinical trials:

1. The gate is a sealed, versioned artifact. The item set, scoring rules, thresholds, judge configuration, and pass/fail logic are bundled and content-addressed. A gate version is identified by a fingerprint of its contents; if any byte changes, it is a different gate with a different identity. There is no such thing as editing gate v7 — only creating v8.
2. The gate is pinned before training begins. When a fine-tuning run is launched in Model Forge, it declares the gate version it will be judged against. This is pre-registration: the criteria are committed before any candidate results exist, so thresholds cannot be set by looking at where a candidate landed. A run cannot retroactively re-point at a friendlier gate; the binding is part of the run's identity.
3. Gate changes are prospective only. A new gate version applies to runs launched after it exists. It can never re-adjudicate a past failure into a pass. If a model failed v7 and v8 relaxes the relevant criterion, the model must be re-run against v8 as a new, fully recorded attempt — the v7 failure remains on the ledger forever.
4. Every attempt is recorded append-only. Model fingerprint, gate version, per-criterion scores, verdict, timestamp. Failures are as permanent as passes. This closes the quiet loophole of "re-run until green": you may re-run, but the ledger shows seventeen attempts, and seventeen attempts against a stochastic judge is itself a signal that reviewers can see and weigh.

Under these rules, arguing with a gate is still possible — but the argument's only lawful output is a proposal for the next gate version, reviewed through the same change-management process as any other production change, with the diff between versions explicit and permanent. In our shop that means the gate change ships with a Codex entry — the executable blueprint/playbook/runbook triple that accompanies every change in our system — so the rationale for v7→v8 is itself a governed, browsable artifact, linked in the dependency graphs at ticket.lucaexpress.com to every model that was ever judged by either version. Ratcheting still can happen, but it happens in daylight, one reviewable version bump at a time, and the longitudinal record that makes drift visible exists by construction.

Anatomy of a gate

A mastery gate in our system is layered, cheapest and most objective first:

candidate model (content-addressed weights + adapter lineage)
        |
  [ L0: structural checks ]      deterministic: schema conformance,
        |                        refusal behavior, format contracts
  [ L1: deterministic scoring ]  versioned scoring engine: exact-match,
        |                        programmatic checks, property tests
  [ L2: evidence-gated AI pass ] model-judged criteria, admissible only
        |                        with extractable supporting evidence
  [ verdict + attempt record ]   append-only ledger entry, either way

L0 answers "is this even a candidate": does the model emit outputs that downstream contracts can parse, does it refuse what it must refuse, does it stay inside its declared envelope. These are binary and uncontestable by design.

L1 is where most of the discriminative power lives. We deliberately push as much evaluation as possible into deterministic, versioned scoring engines — code, not model judgment — because deterministic scores cannot be argued with item-by-item. A property test either holds or it doesn't. The scoring engine itself is versioned inside the gate artifact, so "the scorer changed" can never silently change what a gate measures.

L2 exists because some qualities we care about — faithfulness to a source, tone under a persona contract, reasoning coherence — resist programmatic scoring. Here we use model-based judging, but with a constraint we consider essential and which we hold patent-pending work around: a judge's verdict is admissible only when it is evidence-gated — the judge must cite concrete, mechanically extractable evidence from the model's output (spans, structural features, verifiable claims), and a deterministic layer checks that the cited evidence actually exists and actually supports the ruling. A judge that says "this answer is unfaithful" without pointing at a checkable discrepancy produces no verdict at all. This does not eliminate judge bias, but it converts free-form vibes into auditable rulings, and it makes judge drift detectable: when a new judge version rules differently on the frozen calibration set embedded in the gate, the disagreement is visible as data rather than folklore.

Two further structural choices matter. First, sealed holdouts: gate item sets are drawn from data quarantined at capture time, before it can leak into any training corpus — in Model Forge, quarantine is a property of the capture pipeline, not a discipline we ask engineers to remember. Second, threshold semantics are part of the gate, including tie-breaking, retry budgets, and how stochastic-judge variance is handled (multiple judge samples with a required agreement level, decided a priori). If it isn't written into the sealed artifact, it isn't part of the gate — which means it cannot be invented in the heat of a release.

Trade-offs and failure modes, honestly

We would mislead you if we presented this as free. The costs are real and some tensions are unresolved.

Goodhart pressure is the central tension. A frozen, fully visible gate is a fixed target, and fine-tuning is an optimizer; over enough iterations, training pressure finds the gate's blind spots. The classical mitigation — rotating hidden test sets — is in direct conflict with immutability and pre-registration. Our compromise: the gate's criteria and thresholds are frozen and known; a portion of the item set is sampled at attempt time from a sealed, growing pool that training pipelines cannot read. You know exactly what will be measured and how; you cannot memorize the instances. This is a compromise, not a solution — the pool's distribution is itself a fixed target at one remove — and we treat gate-overfitting audits (comparing gate performance to fresh post-deployment samples) as a standing obligation rather than a solved problem.

Ossification is the mirror-image risk. Make gates hard enough to change and teams stop changing them, and you end up rigorously enforcing criteria that no longer describe the job. Stale rigor is quiet rot. Our countermeasure is procedural: gates carry review clocks, and post-deployment evaluation feeds a standing comparison between gate verdicts and production outcomes. When they diverge, that divergence is the evidence that justifies the next gate version — through the front door.

Immutability moves conflict; it does not remove it. The arguments that used to happen over a failing run now happen over gate-version proposals. That is the point — arguing about the next version is legitimate; arguing about this verdict is not — but it means gate governance needs real ownership and real throughput, or it becomes a bottleneck that teams route around. A gate process people route around is worse than no gate, because it launders bad models through the appearance of rigor.

Cost and latency are nontrivial. Multi-sample judging, evidence verification, and full-ledger provenance make each attempt slower and more expensive than "run the eval script." For small-model fine-tuning at Model Forge scale this is acceptable; we would think carefully before applying the full apparatus to rapid exploratory iteration, and in practice we run a lightweight advisory harness during exploration and reserve the sealed gate for serving eligibility. The gate is a border checkpoint, not a training-loop metric.

Determinism has limits. Serving stacks introduce nondeterminism (batching, kernels, sampling), so "the same model" can score differently across attempts. We pin what can be pinned, sample what cannot, and encode the acceptance statistics in the gate — but anyone claiming perfectly reproducible LLM evaluation is overclaiming, and so we don't.

Why this earns its cost

The payoff is that a passing verdict means something — to us, and to anyone auditing us. Because every serving model carries an unforgeable chain — weights fingerprint → attempt record → sealed gate version → gate's Codex entry and its position in the knowledge graph — the question "why is this model allowed to serve?" has a mechanical answer, years later, independent of who is still at the company. The same discipline that makes our multi-agent delivery pipeline safe to run — serialized merge queues, release trains, changes that ship only with their governed documentation — applies unchanged to model weights, which are, after all, just another artifact that can break production. We think of mastery gates as the merge queue for cognition.

Where this is heading

Three directions occupy our current research. First, gate synthesis: deriving candidate gate criteria semi-automatically from production incident data and Codex runbooks, so gates grow from observed failure modes rather than committee imagination. Second, adversarial gate hardening: red-team passes that attack a proposed gate before it is sealed, searching for outputs that pass the letter of every criterion while violating its intent — cheaper to find these before freezing than after. Third, cross-model gate portability: expressing criteria at a level of abstraction where the same sealed gate can adjudicate a fine-tuned small model and its frontier-model fallback, making "is the small model good enough to take this traffic" a gate verdict rather than a judgment call.

We take on external research and engineering engagements in evaluation infrastructure, fine-tuning pipelines, and AI governance systems — if you are building serving gates you intend to still trust in two years, we would enjoy comparing notes.