A score is a promise. When a number decides whether a change ships, whether a grant application advances, or whether an agent's work is accepted, the number is implicitly claiming three things: that it was computed the same way as every other number it will be compared against, that someone could reconstruct why it has the value it has, and that a low score means the artifact is deficient rather than that the scorer was confused. Most LLM-based evaluation breaks all three promises at once, and does so quietly.
This page describes the scoring architecture we run in production inside Luca AI Express — a deterministic, versioned scoring core with a strictly bounded AI pass layered on top — and the design principles that fell out of building it: the AI may only subtract points, only with verbatim evidence; every score is stamped with an engine version; and when the engine cannot see enough to judge, it caps the achievable score visibly instead of emitting a silent zero. The ideas are simple. What we think is worth writing down is why each constraint exists, because each one was purchased with a specific failure we observed.
The problem with LLM-as-judge as usually practiced
The naive pattern — hand the artifact and a rubric to a model, ask for a number from 0 to 100 — fails along well-documented axes. Scores are not stable across runs even at greedy decoding, because serving stacks are not bitwise deterministic and small logit perturbations flip discretized outputs. Scores drift when the underlying model is silently upgraded. Models exhibit leniency bias, position bias, verbosity bias, and self-preference bias, all reported repeatedly in the LLM-as-judge literature. And most damaging for institutional use: the justification a model gives for a score is a narrative, generated alongside the number rather than causally producing it. You cannot audit a vibe.
None of this means language models are useless for evaluation. It means the architecture must be arranged so that the model's strengths — reading comprehension, pattern recognition over unstructured text, noticing what a checklist misses — are harnessed while its weaknesses — fabrication, inconsistency, sycophancy — are structurally contained rather than merely discouraged by prompt engineering. Prompts are pleas; architecture is enforcement.
Architecture: a deterministic core, an adversarial AI pass
Our engines split scoring into two phases with an asymmetric trust relationship:
artifact ──► [ Deterministic core ]───► base score + cap ladder
versioned rubric │
pure functions ▼
artifact ──► [ AI deduction pass ]────► proposed deductions
subtract-only │ (each with verbatim quote)
▼
[ Evidence gate ]────────► verified deductions only
mechanical check │
▼
final = min(base − Σ deductions, active caps), floored at 0
record = (score, engine version, evidence set, caps hit)The deterministic core computes a base score from checkable structural features of the artifact — properties a pure function can evaluate: presence and completeness of required components, cross-reference integrity, internal consistency conditions, quantifiable coverage measures. Same artifact in, same score out, forever, on any machine. This core carries the comparability promise: two artifacts scored by the same engine version are scored by literally the same function.
The AI pass then plays a single, narrow role: adversarial reviewer. It reads the artifact and proposes deductions — claims that something specific is wrong, each bounded by a per-category maximum defined in the versioned rubric. It cannot award points. It cannot adjust the base. It cannot invent categories. Its entire influence on the number is subtractive and capped.
The evidence gate sits between the AI's proposals and the score. Every deduction must carry a verbatim quotation from the artifact under evaluation — the exact text the deduction is about. The gate mechanically verifies that the quoted span actually occurs in the artifact. No match, no deduction. The check is string matching, not judgment; it requires no model and no trust.
Why subtract-only
The asymmetry is the heart of the design, and it follows from an asymmetry in what hallucination costs in each direction.
If a model can add points, a hallucinated strength — "the document thoroughly addresses rollback procedures" when no such section exists — inflates the score, and nothing downstream can catch it, because the model is asserting the presence of something and you would have to re-read the artifact to falsify the claim. Hallucinated generosity is invisible. Worse, it compounds with the leniency and sycophancy biases these models already carry: the failure mode of an additive judge is systematic, correlated inflation.
If the model can only subtract, hallucination points the other way — a fabricated deduction makes the score too low — and, critically, a fabricated deduction is now checkable, because the evidence gate demands the model put its finger on the exact text. A model cannot verbatim-quote text that does not exist; when it tries, the string match fails and the deduction is dropped. The gate converts hallucination from silent score corruption into a detectable, discardable event.
The residual risk inverts: the AI pass can fail to notice a real problem, and the score comes out too high relative to an ideal reviewer — but never higher than the deterministic base, which was earned through checkable structure. We accept that bias direction deliberately. An engine whose errors are bounded, directional, and auditable is worth more to an institution than one whose errors are smaller on average but unbounded and invisible. This is the same reasoning that makes conservative static analyzers useful: false negatives are a coverage problem you can measure and improve; false positives that silently change decisions are a trust problem you cannot recover from.
Why verbatim, not paraphrase
We gate on exact quotation rather than semantic match for a reason that only becomes obvious after you try the alternative. A semantic-similarity gate is itself a judgment call — you have reintroduced a model into the trust boundary, and now the gate can be wrong in unfalsifiable ways. A verbatim gate is a decidable predicate. It also has a useful side effect on the AI pass itself: models forced to quote before deducting read more carefully. The quote requirement functions as a grounding constraint, the evaluation-side analogue of citation-required generation in retrieval-augmented systems.
The cost is real: legitimate deductions get rejected when the model paraphrases, mis-transcribes whitespace, or quotes across a formatting boundary. We lose some true findings at the gate. But every loss moves the score up toward the deterministic base — the conservative direction — and the rejection rate is itself a measurable signal we track per engine version. When a rubric category shows chronically high gate rejection, that usually means the category is asking the model to judge something that has no textual anchor, which is a rubric bug, not a model bug.
Engine versioning: a score means nothing without its engine
A score is only interpretable relative to the function that produced it. So every engine — rubric, category weights, deduction bounds, cap ladder, and the deterministic core's logic — carries an explicit version, and every emitted score is stamped with it. The record that persists is not a number; it is a tuple: score, engine version, the verified evidence set, and which caps were active. That record is immutable. If the engine changes, we do not edit history — we re-score under the new version and both records coexist, comparable within their own version lineage and explicitly incomparable across it.
This sounds like bookkeeping until the first time someone asks "why did this artifact score 71 last quarter and 64 today?" and the honest answer is "the rubric changed, and here is exactly how." Versioning turns that from an archaeology project into a diff. In our shop this discipline plugs directly into the Codex, our institutional knowledge system: an engine version ships like any other change, with an executable blueprint/playbook/runbook triple seeded into the governed corpus, and its relationships to the artifacts it scores are browsable in the dependency graphs at ticket.lucaexpress.com. The scoring engine is not a script someone owns; it is a governed artifact with a lineage, which is what allows anyone in the organization — human or agent — to contest a score with a reference to the exact rules in force when it was issued.
Versioning is also what makes calibration tractable. Judge-model upgrades are pinned per engine version, so "the model got stricter" and "the rubric got stricter" are never confounded. When we do promote a new judge model, we replay a fixed reference set of artifacts under both configurations and examine the deduction deltas category by category, evidence quote by evidence quote — something that is only possible because deductions are discrete, bounded, and grounded, rather than folded into one opaque number.
Graded caps over silent zeros
The subtlest failure mode in automated scoring is not wrongness but ambiguity of absence. When an engine cannot evaluate something — a required input is missing, a section is unparseable, a dependency wasn't provided — the lazy behavior is to score the affected categories zero. But a zero from "evaluated and found completely deficient" and a zero from "could not evaluate" are different facts wearing the same number, and downstream consumers will average them together, rank on them, and gate on them as if they were the same.
Our engines respond to un-evaluability with graded caps: the condition does not zero a category; it imposes a labeled ceiling on the maximum achievable total, and the ceiling is reported in the score record alongside which condition triggered it. A score of 60 that reads "capped at 60: section X absent" tells the artifact's author exactly what to fix and tells every downstream consumer that this number is a ceiling-limited measurement, not a full evaluation. The cap ladder is itself part of the versioned rubric — caps are graded by severity of the blind spot, and multiple caps compose by minimum, not by multiplication, so a badly degraded input produces a low-but-explained score rather than a collapse to zero through compounding penalties.
The principle underneath is one we apply across the platform: degrade legibly. A system that cannot do its job should say so in its output's shape, not hide the failure inside a plausible-looking value. Silent zeros are the scoring-engine equivalent of a service returning an empty list instead of an error — technically a response, actually a lie.
Failure modes we watch, candidly
- Under-deduction. A subtract-only judge that finds nothing is indistinguishable from a lazy one. We monitor deduction density per category over time; a pass that goes quiet is investigated like a sensor that flatlines.
- Gate attrition. Verbatim matching discards paraphrased-but-real findings. Bias is conservative by construction, but the attrition rate is tracked, and persistent attrition drives rubric revisions.
- Rubric ossification. Deterministic cores reward what they can check. Left alone, authors optimize for the checkable and the rubric slowly measures compliance rather than quality. The AI deduction pass exists partly as a hedge against this — it can notice failures the structural checks cannot express — but rubric review has to be a scheduled practice, not an aspiration.
- Version sprawl. Every rubric tweak forks comparability. We batch changes into deliberate version bumps rather than continuous drift, accepting slower rubric evolution as the price of longitudinal meaning.
- Determinism theater. Temperature 0 is not determinism. We treat the AI pass as nondeterministic on principle, which is precisely why it is only allowed bounded, evidenced, subtractive influence — the deterministic core, not the model, is the part that carries reproducibility.
Where this is heading
The bounded-deduction structure turns out to be an excellent shape for small models. Because the AI pass's job is narrow — read, find, quote, categorize — it is a natural target for the fine-tuned small models we produce through Model Forge, trained on the accumulated corpus of gate-verified deductions; the evidence gate then serves as an online correctness check on the smaller judge, making the swap measurably safe rather than hopeful. We are also extending cap ladders toward confidence-weighted caps, and studying deduction-set stability across judge families as a calibration metric. Aspects of this scoring architecture are covered in our patent-pending portfolio.
Elements of this design — the trust asymmetry, the mechanical evidence gate, versioned score records, legible degradation — transfer to any setting where a model's judgment must be admissible in front of people who are allowed to say "prove it."
Gus IT Research Institution takes on external research engagements in evaluation architecture, AI workflow design, and production LLM systems — reach us through isalabresearch.com.