When AI coding agents author most of a codebase, the economics of software delivery invert. Code generation stops being the scarce resource; verified deployment becomes the bottleneck, and the deploy pipeline becomes the primary quality instrument of the whole system. In Luca AI Express — an AI operating system with a kernel of roughly 49 modules under strict contracts, shipped by several AI agents through a serialized merge queue and scheduled release trains — we learned that most of what makes multi-agent development safe happens not in review, but at the deploy boundary.
This page describes the deploy discipline we converged on: pre-deploy gates, post-deploy smoke verdicts, a hold-not-rollback policy, image pre-warming, deploy-authority locks, and self-healing controllers that know when not to heal. Every rule here exists because a specific production failure taught it. We present the rules and the failures together, because the failures are the argument.
Why the deploy boundary is the control point
Human review of agent-authored code is probabilistic. A reviewer — human or AI — samples a diff and forms a belief. That is acceptable for a single team shipping a few changes a day; it does not compose when multiple agents produce changes continuously against one codebase. Somewhere, probabilistic confidence has to be converted into a binary decision: this artifact runs in production, or it does not.
We place that conversion at the deploy boundary, and we make it deterministic. Diffs are opinions; artifacts are facts. The pipeline interrogates the artifact and the live environment, not the change description, because agent-authored changes are precisely the ones where the description and the behavior can quietly diverge.
merge queue ─► release train ─► PRE-DEPLOY GATES ─► rollout (lease held)
│ artifact liveness │
│ live-state diff ▼
│ schema/migration gate SMOKE VERDICT
│ contract conformance pass │ fail/ambiguous
│ codex-entry presence ▼
▼ HOLD
any fail ⇒ train holds freeze · page · fix forwardPre-deploy gates: diff against reality, not against the repo
The gates that matter least are the ones most CI systems run: does it compile, do unit tests pass. The gates that matter most compare the declared desired state against the live production state, immediately before rollout.
The failure that taught this: a gateway service was redeployed from its manifest during an infrastructure restore, and the manifest was stale — it lacked persistent-volume mounts that had been added to the live deployment months earlier. The rollout was "successful." The service came up cleanly, with amnesia. Nothing in the change had touched storage; the deploy itself was the destructive act, because it reconciled production toward an incomplete declaration. We have since seen this failure class recur in independent subsystems, which is what promoted it from incident to rule: before any redeploy, diff the live object against the declaration being applied, and treat unexplained deletions as gate failures. Redeploys are not idempotent restatements of the present; they are rewrites of it.
Our standing gate battery, conceptually:
- Live-state diff — the rule above, applied to workloads, mounts, and environment surface.
- Schema and migration gate — database changes are staged, rehearsed against a shadow, and must be reversible-or-explicitly-declared-irreversible before the train may carry the code that depends on them. Code and schema ride the same train or the train does not leave.
- Environment-declaration gate — every configuration key a service reads must be declared and present in the target environment. The failure here was a service that behaved perfectly except for one feature whose config had never been promoted; nothing crashed, the feature just silently did nothing.
- Contract conformance — kernel modules ship against explicit module contracts; a build that changes a contract surface without versioning it is rejected mechanically, not editorially.
- Codex-entry presence — under our institutional mandate, no change ships without its executable knowledge entry: a blueprint (what and why), playbook (how to operate it), and runbook (how to recover it), seeded into a governed corpus of 440+ entries. This gate looks bureaucratic until the first 3 a.m. incident where the runbook for the exact failing subsystem already exists because it could not not exist.
A note on gate sociology: gates accumulate, and flaky gates are worse than no gates, because they train agents — and humans — to seek bypasses. Our rule is that bypassing a gate must be louder than passing it: bypass is possible, logged, attributed, and surfaced in the same knowledge graph (browsable at ticket.lucaexpress.com) that tracks the change itself.
Post-deploy smoke verdicts: a deploy is a hypothesis
Orchestrator success — "rollout complete," pods ready — is a statement about scheduling, not behavior. We treat every deploy as a hypothesis and require an explicit verdict: a small battery of behavioral probes that exercise the service's externally meaningful functions and produce a recorded pass/hold/fail object in the deploy ledger. No verdict, no completion; the train does not advance on the absence of bad news.
The teaching failure: a redeploy whose rollout was green in every infrastructure metric, but whose provisioning step had overwritten a safety-critical fallback setting configured out-of-band after the previous release. Health checks passed, dashboards were quiet, and the degradation was only visible if you exercised the failure path the setting protected. Only a behavioral probe — one that asks "does the system still do the thing, including the thing it does when other things break?" — catches this class. Readiness probes measure the process; smoke verdicts measure the promise.
Two design points matter. First, verdicts are evidence-gated: we do use AI passes to interpret ambiguous telemetry, but an AI opinion can only escalate scrutiny, never mint a pass — the same discipline we apply in our deterministic scoring engines. Second, verdicts are versioned artifacts. When a probe battery changes, that change rides a train and carries a Codex entry like any other code, because a silently weakened verdict is a silently weakened production guarantee.
Hold, don't roll back
Rollback is the industry's reflex, and for stateless services with immutable images it is often correct. At AI-system scale it is frequently a second incident wearing the uniform of a remediation. Three assumptions behind "just roll back" fail routinely for us:
- The old artifact still exists. Registry lifecycle policies purge images; the version you would roll back to may be bits that no longer exist anywhere (more below).
- The deploy had no side effects. Migrations ran, provisioning steps executed, agent memory and consolidation state advanced. Rolling back the binary does not roll back what the binary did — and re-running an older provisioning path can actively destroy newer configuration, which is exactly the fallback-erasure failure above, in reverse.
- Old code tolerates new state. After a forward migration, the previous version may be the one that crashes.
Our policy is therefore hold, not rollback. On a failed or ambiguous verdict: freeze the train, retain the deploy-authority lock so nothing else moves, page a human, and fix forward — or, when reversal is genuinely warranted, execute it as its own fully gated deploy with its own verdict, never as a one-keystroke reflex. The honest trade-off: for the minority of incidents where rollback would have been clean, hold-not-rollback lengthens time-to-recovery. We accept longer, understood degradation in exchange for never converting one incident into two. In a system where autonomous agents also operate the pipeline, reflexes are precisely what must be rationed.
Image pre-warming: a version you cannot pull is not a version
Two failures taught us that artifact liveness is a gate, not an assumption. First, a registry's automated retention purge deleted every image tag older than its window — platform-wide — which meant that any reschedule, node replacement, or attempted rollback targeting an older tag would fail at exactly the moment the cluster was already unhealthy. Second, a workload pinned to a node that had died could not simply be rescheduled elsewhere, because "elsewhere" meant a cold image pull through degraded infrastructure during the incident.
The rules that fell out:
- Pre-warm before the window. Ahead of any deploy, pull the target image and the current one onto every node that could plausibly host the workload. The pull happens when everything is healthy and nobody is waiting.
- Artifact liveness is a pre-deploy gate. A release train may not carry a version whose image, and whose predecessor's image, cannot be verified as retrievable and warm.
- Retention policy is production configuration. Whoever owns image lifecycle owns a piece of your recovery story; that dependency is explicit in our dependency graph rather than discovered during an outage.
Pre-warming costs storage and bandwidth and is unglamorous. It is also the difference between a thirty-second failover and an incident review.
Deploy-authority locks and self-healing with humility
With several AI agents, human operators, scheduled trains, and reconciliation controllers all capable of mutating production, the question "who is allowed to deploy right now?" needs a mechanical answer. Ours is a single-writer deploy-authority lease per environment: the merge queue serializes code; the lease serializes runtime mutation. Holding the lease is a precondition for rollout; a verdict of hold keeps the lease pinned so that no scheduled train, eager agent, or controller can move the environment while it is being reasoned about.
Controllers deserve special caution. Self-healing reconciliation is genuinely valuable — it repairs node loss, drift, and partial failures without waking anyone — but a controller that reconciles toward stale desired state is an agent of the past, and during an incident it will faithfully undo your hotfix. The class of failure where a "helpful" redeploy reverted live configuration convinced us that controllers must: reconcile toward the ledger's current declaration, never a cached template; defer to holds unconditionally; escalate to a human after bounded retry rather than override; and write every healing action into the same ledger and knowledge graph as deliberate deploys, so that "what changed?" always has one answer surface. A self-healing controller without humility is just an unattended operator with root.
The Codex: deploy discipline as executable memory
The thread through all of this is that every rule is written down as machinery. Each gate, verdict battery, hold procedure, and controller behavior exists as a blueprint/playbook/runbook triple in the Codex, our governed institutional-knowledge corpus, cross-linked by dependency and knowledge graphs. When an incident produces a new rule, the rule ships the way code ships — through the queue, on a train, with a verdict — and the graph records why it exists and what it protects. Deploy discipline held in the heads of operators decays; deploy discipline held in an executable corpus compounds. For a system whose developers are partly AI agents, this is not documentation hygiene — it is how the agents themselves are made to inherit the lessons.
Where this is heading
The frontier for us is unattended ships: trains that leave without a human at the lever. The blockers are exactly the gates that still encode human judgment — irreversible schema changes and environment-surface promotion chief among them — and we are working to make those judgments either mechanically decidable or mechanically deferrable. In parallel, our Model Forge line points at learned verdicts: small, fine-tuned models reading deploy telemetry to propose pass/hold classifications, always evidence-gated so that deterministic checks retain final authority. Parts of this pipeline architecture are patent-pending; the lessons, we think, generalize to anyone letting agents ship software.
Gus IT Research takes on external research and engineering engagements in this space — deploy pipelines for agent-authored systems, verdict design, and institutional-knowledge tooling. If your agents write more code than your humans can gate by hand, we have probably already hit your next incident.