Most fine-tuning failures we see in the field are not modeling failures. The run converged, the loss curve looked fine, the demo impressed everyone. The failure arrives three weeks later, when someone asks a question the team cannot answer: which dataset snapshot produced the weights currently serving traffic, whether the evaluation that justified promotion ran against those exact weights, and what the safe rollback target is now that the base model has also moved. Fine-tuning, as usually practiced, produces artifacts; production requires lineage. This page describes the lifecycle we built inside Model Forge — the fine-tuning subsystem of our Luca AI Express platform — and the specific bookkeeping most teams skip, because skipping it is what turns a working model into an unoperable one.
The unit of deployment is a chain, not a checkpoint
The mental model that breaks first is "the model is the artifact." In a production fine-tuning system there are four artifact classes, and each one can drift independently of the others:
DATASET ──▶ JOB ──▶ WEIGHTS ──▶ SERVED VARIANT
(frozen (recipe: (immutable (routable identity:
snapshot, base + output of weights + runtime
splits, data + exactly one + eval evidence +
provenance) config) job) promotion state)The arrows matter more than the boxes. Each arrow is a derivation that must be recorded at creation time, because it cannot be reconstructed later. A dataset that was "the export from around March" is not a dataset; it is a rumor. Weights whose training job config lives in someone's shell history are weights you can serve but never reproduce. And a served variant that is just "the latest checkpoint behind the endpoint" is a variant you can neither audit nor roll back with confidence.
Our design rule: every artifact is immutable and identity-addressed; every derivation is an explicit, queryable record; and only the last hop — which variant receives traffic — is mutable. Everything upstream is append-only. This is the same discipline mature software delivery applies to source → build → artifact → deployment, transplanted to a domain where the "build" is stochastic, the "artifact" is gigabytes of tensors, and the "tests" are themselves versioned datasets.
Datasets: capture, freeze, and the splits you must never touch
Fine-tuning data in a live system does not come from a one-time labeling effort. It accumulates: production interactions, agent transcripts, human corrections, synthetic augmentation. Model Forge treats capture as a continuous stream and datasets as frozen cuts of that stream. The distinction is essential. The stream grows forever; a dataset is a named, immutable snapshot with a recorded filter, a recorded cut time, and — critically — recorded splits.
Three practices here separate operable systems from research code:
- Snapshot semantics, not query semantics. A dataset defined as "run this query against the capture store" changes meaning every time the store grows. We materialize the cut and give it an identity. The query that produced it is recorded as provenance, not used as the definition.
- Held-out splits are versioned with the dataset and are contamination-tracked. The single most common silent failure in iterative fine-tuning is evaluation leakage: examples from an old eval split flowing back into a new training cut because both were drawn from the same growing stream. The lineage system has to be able to answer "has any record in eval split E ever appeared in a training cut consumed by an ancestor of these weights?" — mechanically, not by tribal memory.
- Provenance per record, not per dataset. When a served model misbehaves on a class of inputs, the question is which training examples taught it that behavior. Per-record provenance (source interaction, transformation applied, human touch points) is cheap to record at capture time and effectively impossible to recover afterward.
None of this is exotic. All of it is tedious. That is precisely why it gets skipped, and why we made it the substrate rather than an optional hygiene layer.
Jobs: the recipe is the reproducibility boundary
A training job in this lifecycle is a declarative record binding exactly one dataset version, one base model identity (including its own version — base models are upstream dependencies that move under you), and one hyperparameter/recipe configuration. The job record exists before the run, and the run either completes and emits weights or fails and emits a terminal state. There is no editing a job; a changed recipe is a new job.
Two consequences follow that we consider load-bearing:
First, reproducibility is scoped honestly. With LoRA-style parameter-efficient tuning on fixed seeds you can get close to bit-level reproduction; across hardware and kernel versions you often cannot. So the guarantee we engineer for is not "identical weights on re-run" but "identical inputs on re-run, and identical evaluation protocol on the output." The job record is the contract that makes a re-run meaningful even when it is not bitwise identical.
Second, the job is where cost governance lives. Fine-tuning small models is cheap enough that teams will do it constantly — which is the point — but that same cheapness produces sprawl. Because every run is a first-class record with a dataset ancestor and a weights descendant, orphaned experiments are visible, attributable, and garbage-collectable by policy rather than by archaeology.
Served variants and evidence-gated promotion
Weights are not servable in our system. A variant is: weights plus a serving configuration plus an evaluation dossier plus a promotion state. The state machine is deliberately boring — candidate, evaluated, canaried, promoted, retired — and deliberately monotone in one respect: a variant never re-enters a state it left. If a promoted variant regresses, you do not "un-promote" it; you promote its predecessor (or a successor) and retire it. History stays append-only, which is what makes the audit trail trustworthy.
The interesting design decision is what gates the transitions. We use the same pattern we apply in our deterministic scoring engines elsewhere in the platform: evidence-gated evaluation. An evaluation is not a score; it is a versioned tuple of (eval dataset version, protocol version, results, and the concrete evidence — per-item outcomes — backing the aggregate). Promotion requires the dossier, and the dossier pins its own inputs. This closes two failure modes at once:
- Stale evaluation: a variant promoted on the strength of an eval run against different weights, or an eval dataset that has since been revised. Pinned identities make this structurally impossible rather than procedurally discouraged.
- Metric drift: when the eval protocol itself improves (it should — eval suites are living artifacts), old scores and new scores stop being comparable. Versioning the protocol means the system knows two scores are incommensurable instead of silently ranking them.
Where AI-judged evaluations participate — and for small specialized models they usually must, because exact-match metrics undersell them — the judge's outputs are treated as evidence to be retained and spot-audited, never as an unrecorded oracle. Deterministic scoring wraps the stochastic judgment; the stochastic part is versioned like everything else. Aspects of this evidence-gated approach are covered by our patent-pending work, so we describe it here at the method level only.
Rollback is a query, then a routing change
Because the chain is immutable and the only mutable operation is traffic assignment, rollback reduces to two steps: identify the last variant whose promotion dossier is still valid, and repoint routing. The first step is the one teams underestimate. "Roll back to the previous model" is ambiguous when the base model was upgraded between variants, when the eval suite changed, or when the previous variant was trained on a dataset later found to be contaminated. A rollback target is only safe if its entire ancestry is still trusted — which is a graph reachability question, answerable in seconds if you kept the lineage and unanswerable in any amount of time if you did not.
This is also where fine-tuning diverges most sharply from ordinary software rollback: reverting weights does not revert the data distribution that motivated the new training run. A rollback buys you stability at the cost of re-exposing whatever gap the retired variant was built to close. Making that trade-off explicit — visible in the same graph, with the motivating dataset cut linked to the variant — is what lets an on-call engineer make it deliberately at 3 a.m. instead of discovering it as a regression the next week.
The bookkeeping most teams skip
Everything above is mechanism. What makes it survive organizational reality is that in our platform, the lifecycle is wired into the Codex — our institutional knowledge system, in which every change ships with an executable blueprint/playbook/runbook triple in a governed corpus (currently 440+ entries). A new promotion gate does not merely exist in code; it ships with the runbook for what to do when it fires and the playbook for how to reproduce the decision. The lineage relationships — which dataset cut fed which job, which weights back which variant, which variant serves which domain — are browsable as dependency and knowledge graphs at ticket.lucaexpress.com, alongside the rest of the platform's change history. The effect is that the fine-tuning lifecycle has no privileged narrator: any engineer, or any of the AI coding agents in our multi-agent delivery pipeline, can reconstruct why the currently-served variant exists from the record alone.
That last point deserves emphasis. In a codebase shipped by several AI agents through a serialized merge queue, "tribal knowledge" is not just risky — it is unavailable, because the population of contributors includes non-humans with no tribe. The bookkeeping is not documentation about the system; it is the interface through which the system is operated.
Trade-offs and failure modes, candidly
- Immutability has a storage bill and a naming problem. Frozen dataset cuts and retained weights accumulate. Content-addressing and structural sharing between cuts blunt the cost, but retention policy is a real design surface, and "keep everything forever" is not a policy.
- The lifecycle adds latency to iteration. A researcher who wants to "just try something" experiences the job-record discipline as friction. We handle this with an explicit experimental tier whose artifacts are lineage-tracked but promotion-ineligible — the discipline scales down, it does not disappear. Teams that instead carve out an untracked fast path invariably find production weights that originated there.
- Evaluation is the weakest link, structurally. Versioning the eval protocol does not make the protocol good. A perfectly bookkept lifecycle will faithfully promote a bad model past a bad eval. Lineage bounds the blast radius and makes the post-mortem tractable; it does not substitute for eval design.
- Base-model churn is exogenous and relentless. Upstream small-model releases invalidate adapter compatibility and shift eval baselines on their own schedule. Treating the base model as a versioned dependency in the same graph helps you see the churn; nothing makes it cheap.
- Monotone state machines occasionally feel bureaucratic. Re-promoting a predecessor instead of un-promoting a regression produces more records than intuition wants. We accept this deliberately: the moment history becomes editable, every downstream audit weakens.
Where this is heading
Two directions occupy us. First, closing the loop between serving and capture: variants whose production behavior automatically proposes the next dataset cut — with human and evidence gates on the proposal, because an unsupervised data flywheel is how models quietly train on their own failure modes. This connects to the wake/attention/sleep memory-consolidation doctrine we use in our cognitive personas: consolidation from short-term experience into long-term competence, applied to weights rather than memory stores. Second, lifecycle-aware multi-model orchestration — when dozens of small specialized variants serve one platform, promotion decisions stop being independent, and the interesting object becomes the fleet lineage graph rather than any single chain.
We take on external research and engineering engagements in this area — fine-tuning lifecycle design, evaluation gating, and small-model operations — through Gus IT Research Institution.