Multi-Agent Continuous Delivery: N Coding Agents, One Codebase

A single AI coding agent with repository access is now a commodity. The interesting engineering problem begins at N > 1: several agents, each generating plausible code at superhuman cadence, all targeting the same production codebase. At that point the bottleneck stops being code generation and becomes integration integrity — the property that the trunk is always in a state someone (human or agent) can reason about, deploy from, and roll back to.

We run this configuration in production. Luca AI Express — an AI operating system with a kernel of roughly 49 OS modules under strict module contracts and about 22 business domains layered above it — is developed and shipped by multiple AI coding agents working concurrently, merging through one serialized queue, and releasing on scheduled trains. This page describes the delivery architecture conceptually: the invariants we chose, the mechanisms that enforce them, and the failure modes we hit in the order we hit them. It is written for people evaluating whether these patterns transfer to their own systems; we believe most of them do.

Why naive parallelism fails

The intuition people bring from human teams — "just do trunk-based development with CI" — degrades badly when the committers are agents. Three properties of agent-generated change break the human-calibrated assumptions:

• Volume and cadence. Agents produce merge-ready branches at a rate that turns "occasional merge conflict" into a constant background process. Conflict probability per merge may be similar to a human team's; conflict arrivals per hour are not.
• Plausibility without global context. An agent's diff is locally coherent and passes local tests, but the agent reasoned over a snapshot of the repository that may be several merges stale. Two agents can each be individually correct and jointly wrong — the classic write-skew anomaly, expressed in source code instead of database rows.
• Silent semantic overlap. Humans coordinate out-of-band ("I'm touching the billing module this week"). Agents don't, unless the system makes coordination structural. The most dangerous collisions are not textual conflicts Git can detect but semantic ones it cannot: two migrations claiming the same sequence position, two changes to one module's contract, two edits to a shared enumeration.

Our design premise, borrowed from transaction processing, is that you get to pick where serialization happens. Optimistic concurrency everywhere upstream — agents work in parallel, freely, on isolated branches — and a single pessimistic choke point at integration, where order is decided once and enforced.

The shape of the pipeline

  agent A ──► branch ──► ticket-scoped CI ──┐
  agent B ──► branch ──► ticket-scoped CI ──┤     ┌──────────────┐    ┌──────────────┐
  agent C ──► branch ──► ticket-scoped CI ──┼──►  │ serialized   │──► │ release train │──► prod
  agent D ──► branch ──► ticket-scoped CI ──┘     │ merge queue  │    │ (scheduled)   │
                                                  │ (rebase +    │    └──────────────┘
       parallel, optimistic                       │  revalidate) │      batched, gated,
                                                  └──────────────┘      reversible
                                                    one at a time

Everything left of the queue is embarrassingly parallel. Everything right of it is strictly ordered. The queue is where optimism is reconciled with reality.

Branch discipline as an agent contract

Branch policy for humans is convention; for agents it is an enforced interface. Every unit of agent work is bound to a ticket, and the branch is the unit of isolation: one ticket, one branch, one reviewable intent. The rules that matter most in practice:

• Short-lived by construction. Branch age is the single best predictor of integration pain. Agents that let branches ripen accumulate drift against a trunk that other agents are advancing continuously; we treat a stale branch as a defect in itself, not merely a risk factor.
• Scoped diffs. An agent asked to fix a defect will, left alone, also reformat neighboring files, upgrade a dependency, and "improve" an adjacent function. Each of those enlarges the semantic collision cross-section with every other in-flight branch. We enforce scope at review time: a diff that wanders outside its ticket's blast radius is split or rejected. Mixed-intent commits are decomposed per task before they reach the queue.
• Module contracts as conflict firewalls. The AIOS kernel's ~49 modules expose strict contracts, and the contract boundary doubles as a concurrency boundary. Two agents editing the internals of different modules cannot semantically conflict if neither touches a contract. Changes that do touch a contract are a distinct, rarer class routed through heavier review. This is the same argument as interface-based ownership in large human organizations, but agents make it load-bearing: the module graph is effectively a static lock table.

A practical note on shared worktrees: when multiple processes (agents, background committers, humans) share a checkout, automatic commit machinery will happily fuse unrelated work into one commit. We learned to commit early and split aggressively; provenance per ticket is worth more than commit tidiness.

The serialized merge queue

The queue is deliberately unclever. Candidate branches enter with green ticket-scoped CI; the queue takes them one at a time, rebases (or merges) each onto the current trunk head, re-runs validation against that exact post-integration state, and only then advances the trunk. The next candidate sees the world the previous one created.

This is the merge-queue pattern popularized by large open-source and industrial systems (bors-style "not rocket science" rule: never merge anything that hasn't been tested against the exact tree it will produce). What changes with agent committers is the emphasis:

• Revalidation is the point, not an optimization. Because agents reason over stale snapshots, the pre-queue CI result is advisory. The authoritative test run is the one against the rebased tree. Semantic conflicts — code that merges cleanly and then fails — are caught here or not at all.
• Deterministic ordering doubles as an audit log. With a single serialized queue, "what changed between these two production states" has exactly one answer. When an incident review asks which agent's change introduced this, the queue's total order plus per-ticket branches makes attribution mechanical. In a fleet of stochastic authors, deterministic integration is what keeps the system explainable.
• Throughput is bounded and that's acceptable. A strict single-lane queue caps merge throughput at (validation latency)⁻¹. The industry's answer is speculative/optimistic batching — test candidate combinations ahead of the head. We have kept the lane strict longer than a human team would, because agent-authored changes fail revalidation for subtler reasons and speculative rollback amplifies confusion. This is a trade we revisit as validation gets faster; it is a conscious purchase of debuggability with latency.

Collision-free migration allocation

Database migrations are the sharpest edge in the whole system, because they are the one artifact where a textual non-conflict is routinely a semantic catastrophe. Two agents on independent branches each add "the next" migration. Git sees two new files — no conflict. The migration runner sees an ordering ambiguity, a duplicated sequence position, or divergent assumptions about the schema each migration will find when it runs. Unlike code, a bad migration mutates persistent state; the failure is not rebuildable from source.

Naive numbering schemes all fail at N > 1 agents:

• Sequential integers collide constantly — every pair of concurrent branches races for the same successor.
• Timestamps collide rarely but interleave dangerously: a migration authored early and merged late can be ordered before migrations it never knew about, so the schema it assumed is not the schema it gets.
• "Fix it at merge" turns every queue entry that carries a migration into a manual intervention, which at agent cadence means constant intervention.

Our approach treats migration identity as an allocated resource, not an authored one. An agent that needs a migration requests a slot from an allocator that hands out identifiers from a single authoritative sequence, at authoring time — so the reservation, not the merge, is the serialization point for schema evolution. The queue then enforces a complementary invariant: a branch may only merge if its migrations still apply cleanly on top of every migration the trunk has accepted since the branch's snapshot, which the rebase-and-revalidate step checks by construction. Conceptually this is ticket-lock allocation applied to schema changes; elements of this coordination layer are covered by our patent-pending work, so we describe it here at the level of the invariant rather than the mechanism. The invariant is the transferable part: make schema-evolution ordering explicit and allocated, never inferred from filenames or clocks.

Release trains

Merging continuously does not mean deploying continuously. Trunk advances with every queue success; production advances on scheduled release trains that batch queued-and-validated work into a named, tagged, rehearsable unit. The train model buys three things that per-merge deployment costs you:

• A stable unit of verification. A train is rehearsed as a whole — the same batch that was validated is the batch that ships, with database gates and environment checks standing between "built" and "live." Gates that are not yet automated hold the train; we have deliberately kept unattended shipping disabled until every gate is mechanized, because a train that can leave the station with an unchecked gate is worse than a slower train.
• A rollback vocabulary. "Roll back the train" is a well-defined operation; "roll back merges 4,182 through 4,209 individually" is not. Trains also interact with artifact retention: a rollback target is only real if the artifact registry still holds the image, which is a policy question (retention windows) as much as a technical one — a lesson we learned the uncomfortable way.
• Scoped risk. Trains have an explicit scope, and exclusion is first-class: certain production surfaces are simply not on the train, ever, by directive. A delivery system for agents needs negative space — declared regions the automation must not touch — as much as it needs positive capability.

The knowledge layer: shipping the "why" with the "what"

The mechanism that makes the rest sustainable is institutional, not mechanical. Under a standing internal mandate, every change ships with its Codex entry: an executable blueprint (what and why), playbook (how to operate it), and runbook (how to fix it at 3 a.m.), seeded into a governed corpus that now exceeds 440 entries and is browsable — with dependency and knowledge graphs — at ticket.lucaexpress.com.

For a multi-agent pipeline this is not documentation hygiene; it is the long-term memory of the delivery system itself. Agents wake with role, memory, and relevant past items; consolidating each shipped change into the corpus is the "sleep" phase of that loop applied to engineering knowledge. Concretely, it changes queue behavior: an agent preparing a change can traverse the dependency graph to discover what its blast radius actually is, rather than inferring it from grep. The graphs are how agents coordinate without talking to each other — coordination through a shared, queryable model of the system rather than through conversation.

What breaks first

In observed order, on a real production system:

1. Semantic merge conflicts — clean merges, broken behavior. First to appear, permanent background noise. Mitigated (never eliminated) by queue-time revalidation and contract firewalls.
2. Migration collisions — the first severe class. Textual tools are blind to them; this is why allocation had to become a first-class service rather than a convention.
3. Scope creep compounding — individually harmless over-broad diffs raising the global conflict rate until the queue's retry traffic dominates. Fixed at review policy, not in the queue.
4. Queue starvation under load — validation latency times arrival rate exceeding one; branches go stale while waiting, fail rebase, re-enter, and amplify the load. The remedy is shrinking validation time and shrinking diffs, in that order.
5. Infrastructure config regression — redeploys silently dropping operational state (mounts, environment, fallbacks) that lived outside the artifact under test. The train's gates now diff live state against declared state before shipping; this class recurs anywhere declared-vs-live drift is possible.
6. Knowledge decay — the slowest and most expensive failure: changes that shipped without their "why," discovered months later when no agent or human can reconstruct intent. The Codex mandate exists because we hit this.

The meta-lesson: with agent committers, every failure mode you tolerate at human cadence arrives at machine cadence. The defenses that survive are the ones that are structural — enforced by the queue, the allocator, the gates, the graph — rather than behavioral.

Where this is heading

Three directions. First, faster validation to relax the single-lane constraint safely — likely speculative batching with conservative fallback, once revalidation is cheap enough that mispredicted batches are recoverable in minutes. Second, tighter closure of the loop between the Codex graphs and agent planning, so blast-radius estimation is a query the agent runs before writing code, not a property the queue discovers after. Third, applying Model Forge to the pipeline itself: small fine-tuned models specialized for queue-adjacent judgments — conflict-risk triage, diff-scope classification — where a large general model is latency and cost overkill.

We take on external research and engineering engagements in this area — multi-agent delivery pipelines, merge-queue and migration-coordination design, and agent-operated codebases. Reach us through isalabresearch.com.