The limiting reagent in fine-tuning small models is not compute and it is no longer algorithms. LoRA-family adapters, distillation recipes, and preference-optimization methods are commodity knowledge; a competent team can reproduce any of them in a week. What cannot be reproduced in a week is a supply of training examples that are true — drawn from the real distribution the model will face — and clean — free of secrets, personal data, and silent label rot. The richest source of such examples is your own production system: every request it serves is a candidate training record, annotated for free by whatever happened next. The problem is that production traces are simultaneously the best and the most hostile raw material in machine learning. They arrive laced with credentials and personal data, entangled with consent obligations, biased by whatever your current model already does, and mutable in ways that quietly destroy reproducibility.
This page describes the data-engine architecture we built inside Model Forge, the dataset-to-serving pipeline of our Luca AI Express platform, and the design decisions we would defend in front of a review board: where capture happens, where redaction must happen, what provenance a record has to carry to survive an audit, and what a dataset lifecycle looks like when "training-ready" is a gated state rather than a folder name.
Why production traces, and why they fight back
A production AI operating system generates traces constantly: model calls with their inputs and outputs, tool invocations and their results, human corrections, downstream success and failure signals. In our case the substrate is an AIOS kernel of roughly forty-nine modules with strict contracts, plus cognitive personas running a perceive–reason–plan–act–reflect loop; both emit structured events at every stage boundary. That structure matters more than volume. A raw chat log is weak training material because the supervision signal is ambiguous. A trace that says "the planner proposed X, the act stage executed it, the reflect stage graded the outcome, and a human later overrode the result" is a labeled example with a built-in reward signal.
But four properties make traces hostile:
- They contain things you must never train on. Secrets, tokens, personal identifiers, customer content outside its consent scope. One leaked credential memorized by a model is an incident, not a data-quality issue.
- They are distributionally self-serving. Traces show what the current system did, oversampling its habits and undersampling exactly the hard cases where it failed or was never invoked. Naive trace training is imitation learning on your own bugs.
- They are label-ambiguous. "The user didn't complain" is not a positive label. Deciding what counts as a successful episode is a modeling decision, and it drifts as the product changes.
- They are operationally mutable. Upstream services get redeployed, schemas evolve, retention jobs delete rows. A dataset defined as "a query over production" is a dataset that changes under your feet.
A data engine is the machinery that resolves these four tensions deliberately instead of accidentally.
Architecture: capture, quarantine, curation, release
The engine is a staged pipeline in which records only move forward through explicit gates, and every gate is a place where a record can be rejected with a recorded reason.
production quarantine curation release
┌──────────┐ ┌──────────────────┐ ┌───────────────┐ ┌──────────────┐
│ trace │──▶│ raw capture │──▶│ dedup, label, │──▶│ versioned, │
│ emitters │ │ + redaction pass │ │ balance, score │ │ frozen splits │
└──────────┘ │ + consent check │ │ + human review │ │ + eval holdout│
└──────────────────┘ └───────────────┘ └──────────────┘
no train access ◀──── boundary ────▶ trainable material onlyCapture is a first-class concern of the emitting module, not an afterthought of the trainer. Each kernel module's contract includes what it emits into the capture stream and at what granularity — episode-level for agent loops, turn-level for conversational surfaces, call-level for scoring engines. Capturing at the stage boundaries of the cognitive loop, rather than at the network edge, is what makes later labeling tractable: the trace already knows which text was perception, which was plan, and which was action.
Quarantine is where the raw record lives until redaction and consent checks pass. The critical property is that nothing downstream — no curation tool, no notebook, no training job — has read access to quarantine. Redaction that can be bypassed by an impatient engineer with database credentials is a policy, not a boundary.
Curation operates only on redacted material: deduplication, episode reconstruction, labeling, difficulty and quality scoring, and distribution balancing. This is where humans enter, reviewing samples and adjudicating ambiguous labels.
Release freezes a dataset version: immutable membership, fixed splits, a held-out evaluation slice that never touches training, and a manifest recording exactly which curation logic produced it.
Redaction and consent are different problems
Teams routinely conflate redaction (removing content that must not appear anywhere) with consent scoping (respecting what a data owner permitted). They fail differently and need different machinery.
Redaction is a content-level transformation and it must be layered, because every individual layer misses things. We compose deterministic detectors for structured secrets (they have shapes; pattern matching plus entropy heuristics catch them reliably), model-assisted detection for unstructured personal data (names, addresses, biographical detail that no regex will find), and format-preserving replacement rather than deletion — substituting realistic placeholders so the record stays syntactically and semantically trainable. A redacted-to-ellipsis dataset teaches the model that ellipses are normal output. The honest engineering position is that recall is never 1.0, which is why redaction sits before the trainable boundary and why we sample released data for leaked-content audits continuously rather than trusting the pass that ran at capture time.
Consent is a record-level property, not a content transformation, and its defining nightmare is revocation. If a data source's terms change, or a customer exercises a deletion right, you must be able to answer: which released datasets contain records from this source, and which trained models consumed those datasets? That question is unanswerable retroactively. It is only answerable if consent scope was attached to every record at capture time and propagated through every derivation — which is a provenance problem.
Provenance: the lineage graph is the product
Our strongest position in this whole design is that a training record without provenance is a liability, not an asset. Every record in the engine carries, from capture onward: its originating source and consent scope, the capture context (which module, which pipeline version, when), the full chain of transformations applied to it, and the identity of every dataset version it was released into. Every dataset version, in turn, records its curation logic version and its parent datasets; every fine-tuned model records its exact dataset versions.
This forms a directed graph from source → record → dataset version → training run → model → deployment, and we treat it as a first-class queryable artifact rather than scattered metadata. It is the same discipline we apply to code: in our institutional knowledge system, the Codex, every change ships with an executable blueprint/playbook/runbook triple into a governed corpus, browsable as dependency and knowledge graphs at ticket.lucaexpress.com — and dataset releases are Codex entries like any other change, with their lineage rendered in the same graph tooling. The payoff shows up in exactly three moments, all of them bad days:
- Revocation: walk the graph from a source to every affected model, and know precisely what must be retrained or retired.
- Regression forensics: a fine-tuned model develops a behavioral quirk; walk backward to the dataset delta between the good and bad versions, then to the specific curation change or trace cohort that introduced it.
- Audit: demonstrate, with records rather than assertions, that no training run consumed data outside its consent scope. Parts of our provenance and lifecycle machinery are covered by patent-pending work, so we describe it here at the level of method rather than mechanism.
The lifecycle: "training-ready" is a state you earn
A dataset in the engine is a state machine, not a directory:
CAPTURED → QUARANTINED → REDACTED → CURATED → RELEASED → CONSUMED
↘ DEPRECATED → RETIREDThree rules give the lifecycle its teeth. First, forward-only gates: a record cannot skip a state, and each transition is performed by tooling that logs its reasons — there is no manual path from raw capture to a training set. Second, immutability at release: a released version never changes; corrections produce a successor version with a recorded diff, which is what makes training runs reproducible and evaluation comparisons meaningful. Third, deprecation is an active state: when a release is superseded — for a redaction miss, a consent change, or discovered label rot — consumers are notified through the lineage graph, and retirement (actual deletion, honored through derived artifacts) is a tracked operation rather than a hopeful email.
The under-appreciated stage is the loop closure: models trained on released datasets go back into production, their traces flow into capture, and the engine is now feeding on its own outputs. Without countermeasures this compounds bias generation over generation. We treat it with cohort tagging (records trace which model generation produced them, so training mixes can cap self-generated fractions), deliberate oversampling of failure and human-override episodes, and evaluation holdouts drawn from human-origin data that never rotates into training.
Trade-offs and failure modes, candidly
Redaction recall versus data utility. Aggressive redaction destroys the specificity that makes production data valuable; conservative redaction leaks. There is no setting that eliminates the trade-off — only a boundary placement (quarantine before trainability) that makes the residual risk auditable, plus continuous sampling audits that measure it.
Curation logic is code, and it rots. A heuristic that filtered low-quality episodes correctly last quarter silently misfilters after a product change. Because curation versions are recorded in provenance, this is detectable; it is still not free. Budget for curation regression review the way you budget for test maintenance.
Human labeling is the bottleneck and the drift source. Reviewer standards shift over months. We mitigate with adjudication sampling and by keeping labeling guidelines as versioned Codex artifacts, but anyone claiming label consistency is solved is selling something.
Provenance overhead is real. Carrying lineage through every transformation costs storage, engineering discipline, and pipeline complexity. Our judgment is that the cost is an order of magnitude smaller than a single unanswerable revocation request, but teams should adopt it before their first fine-tune, not after — retrofitting provenance onto an existing dataset estate is close to impossible.
The feedback loop is never fully closed. Cohort caps and human-origin holdouts bound self-training drift; they do not eliminate it. Periodic injection of genuinely external evaluation material is the only honest control.
Where this is heading
The direction of travel is toward the data engine as the primary asset and models as its disposable outputs. Concretely, we are extending the engine in three directions: tighter coupling between our deterministic, evidence-gated scoring engines and the labeling stage, so that outcome grades become training signal without a human in every loop; consent-scope-aware training mixes, where a single run can prove per-batch that every record was within scope; and memory-consolidation-derived data — our personas' wake/attention/sleep consolidation cycle already distills episodic traces into durable knowledge, and that distillation output is itself a promising curated-dataset source. The unifying bet is that small, frequently retrained models on top of a rigorously governed data engine beat large, rarely retrained models on top of a data swamp — and that the governance is precisely what makes frequent retraining safe enough to do.
Gus IT Research takes on external research engagements in this area — data-engine design, fine-tuning pipelines, and dataset governance — for teams who want this machinery without learning its failure modes the expensive way.