Serving Small Models on Modest Hardware

How entry-level datacenter GPUs, careful quantization, and a locally scheduled inference pool turn small-LLM serving from an API line item into owned infrastructure with predictable economics.

There is a quiet assumption in most inference planning conversations: that serving language models means either renting frontier-model APIs by the token or standing up eight-way H100 nodes. For a large class of production workloads, both assumptions are wrong. Once a task has been narrowed — classification, extraction, routing, scoring, structured drafting, retrieval synthesis over a bounded corpus — a fine-tuned model in the 1B–15B parameter range frequently matches or beats a general frontier model on that task, and it fits comfortably on hardware that costs less than a mid-range server did a decade ago.

We run Luca AI Express, a production AI operating system whose kernel dispatches work across dozens of modules and business domains. Many of those dispatches are exactly the narrow, high-volume calls described above. Our Model Forge pipeline (dataset capture → fine-tuning → serving → evaluation) exists to convert recurring frontier-API calls into owned small models, and the serving tier it feeds is deliberately built on modest hardware. This page describes what we learned making that tier work: which hardware constraints actually bind, how quantization formats behave in practice, how a local inference pool should be scheduled, and where the economics genuinely break even — and where they don't.

The binding constraint is memory bandwidth, not FLOPs

Autoregressive decoding is a memory-bound workload. At batch size 1, generating each token requires streaming essentially the entire weight set (plus the KV cache) through the GPU's memory subsystem. A rough ceiling for single-stream decode throughput is:

tokens/sec  ≈  memory_bandwidth  /  bytes_per_forward_pass
            ≈  memory_bandwidth  /  (active_params × bytes_per_param + KV traffic)

This has three consequences that shape everything downstream:

Entry-level datacenter GPUs are more competitive than their FLOPs suggest. Cards in the L4/A10/T4 class have modest compute but respectable bandwidth relative to their price. For decode-dominated workloads, the gap to flagship parts narrows dramatically — you are paying for bandwidth and VRAM, and the premium parts' tensor-core superiority sits partially idle.
Quantization is a throughput feature, not just a fit-it-in-memory feature. Halving bytes-per-parameter roughly doubles the decode ceiling. A 4-bit 8B model on a bandwidth-modest card can out-decode a 16-bit 8B model on a substantially more expensive one.
Prefill and decode want different hardware. Prefill (prompt ingestion) is compute-bound and batches beautifully; decode is bandwidth-bound and batches only up to the point where the KV cache exhausts VRAM. Mixed workloads on one card mean one phase is always subsidizing the other, which matters when you schedule a heterogeneous pool.

The practical upshot: for small models, "modest hardware" is not a compromise position. It is often the correct position, provided the scheduling layer above it is doing its job.

Quantization formats, and what actually degrades

The format landscape settled into a few stable families, each with a distinct operating envelope:

Weight-only integer formats (GPTQ, AWQ, and the GGUF k-quant family). These quantize weights to ~4 bits using calibration data while keeping activations in higher precision. AWQ's activation-aware scaling and the newer importance-weighted GGUF variants hold up remarkably well at 4-bit for models above roughly 7B. Below ~3.5 effective bits, quality falls off a cliff rather than a slope — and the cliff's location varies by model family and by how aggressively the model was trained.
FP8 and INT8 weight-and-activation formats. These enable faster GEMMs on hardware that supports them and are the right choice when you have batch-heavy prefill traffic. On entry-level cards without native FP8 paths, their advantage largely evaporates, which is a real selector between GPU generations.
KV-cache quantization. Often the highest-leverage lever nobody applies. Long-context serving is frequently VRAM-bound on cache, not weights; 8-bit cache is close to free in quality, and 4-bit cache is workload-dependent in ways that generic benchmarks will not reveal.

The lesson we would emphasize to anyone evaluating formats: perplexity deltas are not decision-grade evidence. A quantized model can sit within noise on perplexity and still regress meaningfully on the narrow task you fine-tuned it for — instruction adherence, structured-output validity, and rare-label recall degrade earlier than fluency does. Fine-tuned small models are more fragile under quantization than their base models, because fine-tuning concentrates task behavior into weight structure that calibration sets drawn from generic corpora do not exercise.

Our response is procedural rather than clever: every quantized artifact in Model Forge passes through the same evidence-gated evaluation harness as the full-precision model, on the task's own eval set, before it is eligible for the serving pool. Quantization is treated as a model change, not a packaging step. In our Codex — the institutional knowledge system where every production change ships with an executable blueprint/playbook/runbook triple — a quantization recipe is a first-class entry with its own dependency edges, so when a base model or calibration approach changes, the graph at ticket.lucaexpress.com shows exactly which served artifacts are downstream and need re-gating. This sounds like bureaucracy until the first time a silent 4-bit regression ships to a scoring path; after that it looks like the minimum viable process.

The local inference pool

A single GPU serving a single model is easy. The interesting engineering starts when you have a shelf of modest, heterogeneous cards and a catalog of small models whose combined VRAM footprint exceeds the pool. The problem becomes an operating-systems problem — which is congenial territory for us, since Luca's AIOS kernel already thinks in terms of module contracts and scheduling.

            ┌────────────────────────────────────────────┐
 requests → │ router: task → model → placement decision  │
            └───────┬──────────────┬─────────────────────┘
                    │              │
             ┌──────▼─────┐  ┌─────▼──────┐   ┌────────────┐
             │ GPU A       │  │ GPU B      │   │ GPU C      │
             │ model-x(4b) │  │ model-y(4b)│   │ (warming   │
             │ model-z(4b) │  │ kv-heavy   │   │  model-w)  │
             └────────────┘  └────────────┘   └────────────┘
                    ▲
        residency policy: pin hot models, evict cold,
        prefetch on schedule, never thrash

The design decisions that mattered most in practice:

Model residency is the scheduling problem; request routing is easy. Loading a multi-gigabyte model is seconds-to-minutes; a decode step is milliseconds. The pool's job is to make loads rare: pin persistently hot models, co-locate small models on shared cards where VRAM permits, and evict by observed demand rather than LRU alone. We found it useful to think of this in the same wake/attention/sleep terms as our persona memory doctrine — a model that is "awake" holds VRAM, a "drowsy" model's weights stay staged in host RAM for fast re-load, and a "sleeping" model lives on disk. The analogy is not decorative; the same demand-forecasting signals drive both.
Continuous batching with paged KV management is table stakes. The vLLM-style approach — treating KV cache as paged memory rather than contiguous per-request allocations — is what makes modest-VRAM cards viable under concurrency. Without it, worst-case cache reservation forces you to size for the longest possible request and idle the difference.
Separate the latency classes. Interactive traffic (agent tool-calls, user-facing routing) and throughput traffic (batch scoring, dataset generation for Model Forge) should never share a card's decode budget uncontrolled. We route them to distinct pool partitions with different batching parameters; the batch partition also becomes the natural home for opportunistic work when interactive demand dips, which is where most of the utilization recovery comes from.
Speculative decoding earns its complexity only sometimes. Draft-model speculation helps when the target model is large relative to the card's bandwidth and acceptance rates are high. For already-small, already-quantized models, the draft model's overhead and the added operational surface often erase the gain. Measure per task; do not adopt by default.

Failure modes we have paid for: VRAM fragmentation after repeated load/evict cycles forcing periodic worker recycling; driver/runtime stack drift across a heterogeneous fleet (two cards, same nominal setup, different kernels selected, different numerics); and thermal throttling on dense shelves quietly reshaping tail latency in ways that look like software regressions. Each of these now has a runbook in the Codex, because each of them recurred until it did.

The economics of owning your inference

The honest framing: owning inference is a utilization bet. The comparison is not "API price per token vs. electricity per token" — that comparison flatters ownership absurdly. The real ledger includes hardware depreciation over a realistic (short) useful life, power and cooling, the engineering time to run the pool, and the option value you give up by not being able to switch models with a config change.

What we can say from operating this way, without inventing numbers:

Break-even is dominated by sustained utilization, and sustained utilization is dominated by workload shape. Steady, high-volume, narrow tasks — exactly the tasks small fine-tuned models are good at — are the ones that keep a pool busy. If your traffic is spiky and diverse, APIs remain the right answer for the spikes even if you own the base load.
The token-price asymmetry is structural. A task that a frontier API serves at frontier prices is often servable by a 4-bit 8B fine-tune at a per-token marginal cost that is a small fraction of it. Ownership converts a variable cost that scales with product success into a step-function fixed cost. For a platform like ours, where the AIOS kernel generates a large volume of machine-to-machine calls that no human ever reads, that conversion is the whole business case.
The non-financial returns are real but should be priced separately. Data residency (prompts and outputs never leave your infrastructure), latency floors unmediated by a provider's queue, immunity to model deprecation on someone else's schedule, and the ability to fine-tune and re-serve within hours — these justified our pool on days when the pure cost math was merely neutral.
The costs people omit: evaluation infrastructure (owning models means owning the burden of proving they still work), model churn (a better open-weights base ships quarterly, and re-fine-tuning/re-quantizing/re-gating your catalog is recurring labor), and the organizational cost of hardware being a queue rather than an elastic resource.

Our position is deliberately hybrid: frontier APIs for open-ended reasoning and low-volume tasks, owned small-model serving for the narrow high-volume core, and Model Forge as the conveyor that moves tasks from the first category to the second when volume justifies it. The decision of when to move a task is itself evidence-gated — captured traffic, a fine-tune, an eval showing task-level parity, and only then a routing change, with the whole chain recorded as Codex entries so the decision is reproducible and reversible.

Where this is heading

Three directions occupy us currently. First, tighter co-design of quantization and fine-tuning — quantization-aware fine-tuning so the 4-bit artifact is the trained object rather than a lossy export of it, which should move the quality cliff meaningfully. Second, smarter pool scheduling: demand forecasting from the AIOS kernel's own dispatch telemetry, so model residency decisions are made ahead of traffic rather than behind it. Third, extending our evidence-gated promotion pipeline down to the hardware level, so a new GPU SKU or runtime version is qualified for the pool the same way a new model artifact is — as a governed change with a blueprint, a playbook, and a runbook, visible in the dependency graph. Portions of this work are patent-pending.

We take on external research and engineering engagements in exactly this territory — small-model serving economics, quantization evaluation, and inference-pool architecture. If you are deciding whether to own your inference, that is a conversation we have had with ourselves, in production, for some time.