Local LLM Inference Server
A dual-GPU local AI platform serving a 9B–321B model fleet across llama.cpp, Strata, and Project Maya — with a production 125B MoE, tiered expert memory, and engine-aware telemetry.
Project Brief
- Role
- Solo infrastructure owner and operator
- Scope
- A dual-GPU Linux server exposing a 9B–321B OpenAI-compatible fleet across llama.cpp, Strata, and Project Maya, with a production 125B MoE and optional 321B profile, plus authenticated external agent access
- Timeline
- Continuously operated and tuned in 2026
- Result
- A three-engine local AI platform serving MoE models beyond GPU-only memory capacity through tiered experts and KV streaming, with workload-specific operator benchmarks, engine-aware telemetry support, and a separate in-progress draft-model training effort
Evidence Included
Live production telemetry
When the rig is awake and the exporter supports the active engine, the panel reports model, GPU, and inference activity. Strata/Maya engine detail is optional; missing fields or historical counter gaps are not evidence of zero usage.
Measured tuning results
The operator's October 2026 brief reports Strata 125B at ~204 tok/s short decode and ~115 tok/s at 141K context, plus Maya 321B at ~35 tok/s. These context-dependent benchmarks are not independently rerun measurements or current live rates; older MTP and hardware-tuning results remain historical evidence.
Public implementation context
The architecture, model-swapping behavior, capacity constraints, and observability stack are documented in detail on this page.
Live debugging, not just a demo
Two real bugs in the status service itself — a stale model cache and a biased GPU-utilization reading — were root-caused and fixed with this site's own live feed as the test harness.
9B–321B · 3 engines
Model fleet
125B MoE · Strata
Production default
~204 tok/s
Strata short decode (operator benchmark)
~115 tok/s
Strata 141K decode (operator benchmark)
~2,300–2,400 tok/s
Strata 32K prefill (operator benchmark)
~35 tok/s
Optional Maya 321B decode (operator benchmark)
2x 16GB · VRAM / RAM / SSD
GPUs + expert tiers
Up to 256K · KV streaming
Supported-profile context
inference:~$ status --watch
connecting
raw feed ↗ (opens in new tab)establishing uplink…
A home-built local AI infrastructure project: a dedicated Linux server running a dual-GPU (2x 16GB VRAM, PCIe Gen 4) inference rig behind an OpenAI-compatible API. Rather than keeping one model loaded at all times, the stack hot-swaps between open-weight LLMs spanning 9B–321B parameters across llama.cpp, Strata, and Project Maya. The production default is Qwen3.8-Flash-Next 125B on Strata; GLM-5.3-Flash 321B on Project Maya is an optional, memory-intensive alternative, not the default. These sparse MoE models do not fit entirely in GPU memory: expert weights are tiered across VRAM, pinned host RAM, and SSD fallback, with a CPU execution lane and RAM-backed KV streaming — all behind one stable endpoint any OpenAI-compatible client or coding agent can talk to. Every major tuning effort is evaluated with workload-specific benchmarks rather than assumed gains. Operator-reported benchmarks include ~204 tok/s short decode for Strata and ~35 tok/s for Maya; these depend on context, quantization, and configuration, and are not current live speeds. The rig is also self-monitoring: a small read-only, key-authed status API feeds the live panel on this page, and has itself been debugged and hardened in production rather than just built once and left alone — see the highlights below for two real bugs found and fixed in it. It's since grown from an internal-only rig into a public, authenticated AI provider — reachable by external coding agents and OpenAI-compatible tools, not just clients on the home network. A second, separate dual-GPU workstation (192GB VRAM class) now runs alongside it purely for training work: extracting hidden-state data to train a small custom draft model intended to speed up decoding on the original inference rig once validated.
Highlights
- Three-engine model-swapping platform — llama-swap fronts llama.cpp, Strata, and Project Maya behind one OpenAI-compatible API, with a 9B–321B fleet on two 16GB GPUs; changing models unloads the resident engine rather than keeping the whole fleet in VRAM
- Production Qwen3.8-Flash-Next 125B on Strata: operator benchmarks report ~204 tok/s short decode, ~115 tok/s at 141K context, and ~2,300–2,400 tok/s prefill at 32K — workload-specific measurements, not a live-speed guarantee
- Optional GLM-5.3-Flash 321B on Project Maya, a Strata fork: operator benchmarks report ~35 tok/s decode and ~690 / ~1,010 tok/s prefill at 8K / 32K; quantization, expert-promotion tuning, and DFlash2 improved decode from a 23.3 tok/s default baseline by roughly 50%
- Tiered MoE execution keeps hot experts in VRAM, stages others in pinned RAM, falls back to SSD, and uses a CPU lane alongside GPU execution; RAM-backed KV streaming separates total context from the tokens resident in VRAM, with up to 256K context on supported profiles rather than a fleet-wide guarantee
- Speculative decoding is engine- and model-specific: Maya uses a DFlash2 drafter, distinct from MTP on supported llama.cpp profiles; historical MTP speedups and acceptance rates are not claims about the newer engines
- Enabled PCIe Resizable BAR and Above 4G Decoding at the BIOS level, expanding addressable GPU memory-mapped I/O from 256MiB to 16GB per card — cut model load time by roughly 40% and cold model-swap latency by roughly 45%, both validated with repeated before/after timing
- Locked the CPU to its performance governor across all cores specifically to remove prompt-prefill latency variance introduced by frequency scaling under bursty inference load
- Extended the rig beyond the LAN into a public, authenticated AI provider — external coding agents and OpenAI-compatible tools can now reach the fleet from anywhere over an outbound-only tunnel, with zero inbound firewall changes; the raw inference port itself stays exactly as LAN/VPN-only as before
- Fronted that public route with a purpose-built gateway enforcing four independent layers before a request ever reaches a model — bearer-key authentication, an endpoint allowlist (only chat/completion-style routes resolve; admin, metrics, and UI endpoints 404 regardless of auth), a concurrency guard matched to the fleet's real one-request-at-a-time GPU capacity so overlapping requests queue briefly instead of colliding, and per-key rate limiting — plus a second, independently-enforced rate limit at the network edge as defense in depth
- Verified the gateway end-to-end from the public internet before trusting it with real traffic — rejected invalid/missing auth, confirmed admin-style routes stay unreachable even with a valid key, exercised a full real chat-completion round trip, and proved the concurrency queue behaves correctly under overlapping requests rather than erroring
- Host and GPU monitoring through Prometheus and three Grafana dashboards, complemented by an engine-aware status contract for Strata/Maya request state, prefill/decode rates, expert-cache behavior, KV residency, CPU lane, per-GPU memory allocation, and recent requests; fields depend on engine support and exporter availability
- Validated GPU memory overclocking through structured A/B testing — ~17% higher effective memory clock with zero throughput regression across 6 sustained stress cycles and a full reboot re-validation
- Caught a live GPU core-clock change silently collapsing one model's MTP draft acceptance from 65–80% to 0% (and decode speed from 73 to 29 tok/s) — root-caused, reverted, and turned into a standing rule: clock changes only happen at boot, never against a running inference session
- Diagnosed a normalization-equivalence bug degrading one hybrid linear-attention model's output and triggering premature end-of-sequence tokens once context passed roughly 100K tokens, traced it to a known upstream llama.cpp defect (fixed in ggml-org/llama.cpp PR #28068, commit 5fdfa62), and deployed the fix via a custom-compiled build ahead of its stable release rather than working around the symptom or waiting
- Benchmarked a new coding-focused model (Ornith 1.0 35B) against the fleet leader on public benchmarks before adopting it — up 2.2 pts on SWE-bench Verified, 11.7 on Terminal-Bench 2.1, and 5.2 on NL2Repo — and kept it as an A/B candidate pending real-workload validation rather than an outright replacement
- Automated power lifecycle for the rig itself — a Home Assistant-controlled smart plug lets the whole server power on/off on a schedule or on demand, avoiding idle power draw when the fleet isn't in active use
- Self-monitoring by design — a key-authed, rate-limited, CORS-locked status API feeds this page separately from the authenticated public inference gateway; raw engine, administration, and monitoring services remain LAN/VPN-only
- Found and fixed two real bugs in that same status service: a stale in-memory model cache that hid newly added models until a manual restart, and a GPU-utilization reading stuck near 0% because it sampled a bursty, sub-second signal once every 20 seconds — root-caused by polling nvidia-smi directly during a live generation, fixed with config-driven capability detection and sub-sampled averaging, and verified live end-to-end both times
- Replaced the panel's blended cloud-cost estimate with separate input, output, and cached-input accounting; its historical 32B hosted-model reference is illustrative, not a fleet-wide upper bound or verified savings for the newer 125B/321B models. Earlier llama.cpp cache-reuse accounting was checked against request-level figures; newer engines require their own counters
- Extended MTP decoding to a text-only profile of the vision-capable model, and — while chasing an unrelated GPU-overclock question — isolated and documented a reproducible non-determinism quirk specific to that model's vision-encoder path, ruling out hardware instability through controlled, repeatable testing
- Wired into day-to-day developer tooling — coding agents and IDE integrations route requests to the local fleet exactly like a cloud LLM provider
Architecture & Infrastructure
Model-swapping API gateway
A lightweight proxy (llama-swap) fronts llama.cpp, Strata, and Project Maya and exposes one stable OpenAI-compatible endpoint. Requesting a different model by name transparently unloads the current engine, releases its resources, and loads the next — clients never need to know which engine is physically resident. Load time depends on the selected model and its host-memory requirements; historical small-model swap timings do not describe the entire newer fleet.
Tiered MoE memory beyond VRAM capacity
Two 16GB GPUs serve a fleet spanning 9B–321B total parameters, not fully GPU-resident weights. Strata runs the production 125B MoE; Project Maya extends that architecture for the optional 321B model. Hot experts occupy VRAM, pinned RAM supplies another tier, and SSD provides fallback. A CPU lane computes experts alongside the GPU path. Maya's sample profile pins roughly 70GB of host RAM while loaded, so 32GB of VRAM is not the system's total memory requirement.
RAM-backed KV streaming
Strata/Maya distinguish configured context from GPU-resident KV tokens: KV cache streams from RAM instead of requiring the entire context to stay in VRAM. The Maya sample configures 256K context with 32K resident tokens. Context capacity and vision support remain profile-specific; neither implies every model supports 256K or that long-context throughput equals short decode.
Speculative decoding for real throughput gains
Drafting is not synonymous with MTP. Supported llama.cpp profiles use multi-token prediction, while Maya uses the DFlash2 speculative drafter. The engine contract identifies the actual drafter separately from the engine; an MTP flag alone cannot describe the fleet. Historical llama.cpp measurements of ~1.5–2x speedup and 65–80% acceptance apply to those tested profiles, not automatically to Strata or Maya. Draft acceptance is reported only where supported.
Host monitoring and engine-aware telemetry
Prometheus scrapes GPU, host, and inference-server metrics into three Grafana dashboards: a hand-built one tailored to this box, plus two widely-used community imports (Node Exporter Full and NVIDIA GPU Metrics) adopted instead of reinventing well-covered ground — the same build-vs-adopt judgment call expected on a production observability stack. The newer Strata/Maya status contract adds idle/prefill/generation state, expert VRAM hit rates and RAM/SSD traffic, CPU-lane allocation, KV residency, per-GPU memory breakdowns, request history, and process-lifetime counters. These are optional engine-specific fields, not a claim that every engine exposes every metric or that exporter rollout has already been verified here.
Evidence-based performance tuning
GPU memory overclocking was pushed and validated incrementally with repeatable benchmarks (determinism checks, sustained stress runs, full reboot cycles) rather than applied blindly, and an unstable setting was rolled back before it could affect real use. Separately, a live core-clock change — applied while inference processes already held an active CUDA context — silently collapsed one model's MTP draft acceptance from 65–80% to 0% and its decode speed from 73 to 29 tok/s; reverting the clock value alone didn't fix it, only a full service restart did. Root-caused and turned into a standing rule: clock changes only ever happen at boot, never against a running inference session.
Benchmark-driven model evaluation
New models are evaluated against the current fleet leader on public, third-party benchmarks (SWE-bench Verified, Terminal-Bench, NL2Repo) before being promoted — Ornith 1.0 35B beat the incumbent on all three, but ships as a coexisting A/B candidate rather than an outright replacement until it's proven out on real workloads, not just benchmark scores.
Self-monitoring, hardened public telemetry
A small, dependency-free status API on the box itself tracks lifetime token counts, per-model best-observed throughput, and rolling activity history, persisted across model swaps and restarts. Telemetry and the authenticated inference gateway are separate public surfaces: the status API is key-authed, origin-locked, rate-limited, and funneled out over Tailscale rather than a forwarded router port. Raw engines, administration, and monitoring remain LAN/VPN-only. The earlier exporter was llama.cpp-specific; Strata/Maya accounting depends on the engine-aware exporter, and historical gaps must not be treated as zero activity.
Public AI provider gateway
A purpose-built gateway fronts a public, outbound-only tunnel into the fleet so external coding agents and OpenAI-compatible tools can use it too, layering four independent controls in front of every request: bearer-key authentication, an endpoint allowlist (only chat/completion routes resolve — admin, metrics, and UI endpoints 404 regardless of auth), a concurrency guard sized to the fleet's real one-request-at-a-time GPU capacity so overlapping calls queue briefly rather than collide, and per-key rate limiting, backed by a second, independently-enforced rate limit at the network edge. The underlying inference port itself never becomes internet-facing — raw engines, administration, and monitoring stay LAN/VPN-only, with public telemetry exposed separately. Previously verified end-to-end from the public internet — auth rejection, admin-route lockout, a real completion round trip, and correct queuing under overlapping requests — before it carried any real traffic.
Operational debugging, not just uptime
The status service itself has needed real fixes, found and resolved the way any production issue would be. A model roster fetched once at process start and cached forever (so a newly added model stayed invisible until a manual restart) was replaced with a live refresh every poll cycle; a hardcoded vision/MTP allowlist that could silently drift from the real config was replaced with a scan of the config's own launch flags; and a GPU-utilization reading stuck near 0% almost permanently was traced — by polling nvidia-smi directly during a live generation — to a single instantaneous sample of a signal that only spikes for milliseconds at a time, then fixed by sub-sampling and averaging across each reporting window. Every fix was verified against the live public endpoint before and after, with no downtime.
Itemized cost modeling instead of one blended rate
The original “cloud cost avoided” estimate priced every token — prompt and generated alike — at one flat rate, and quietly excluded prompt tokens from the total entirely. Real hosted-inference vendors bill input, output, and cached-input tokens separately, and this fleet now spans 9B–321B models across three engines. The panel's historical reference quote for DeepSeek R1 Distill Qwen 32B and assumed cached-input discount are an illustrative comparison, not a current per-model quote, a fleet-wide upper bound, or net savings after hardware and electricity. Earlier llama.cpp cache reuse was reconstructed from completed requests and checked against request-level figures. Strata/Maya require their own prompt, reused, and output-token accounting; unsupported or missing counters limit the estimate rather than proving no usage occurred.
PCIe Resizable BAR for faster model loads and swaps
Enabling Resizable BAR and Above 4G Decoding at the BIOS level expanded each GPU's addressable memory-mapped I/O window from the default 256MiB to 16GB, letting the CPU map a full model's weights for direct access instead of moving them through a narrow aperture in smaller chunks. Measured before/after: model load time dropped by roughly 40%, and — more operationally relevant for a fleet that hot-swaps models constantly — cold model-swap latency dropped by roughly 45%.
Speculator training on a separate GPU workstation
A second, separate dual-GPU workstation with a much larger VRAM budget runs independently of the production inference rig, dedicated to training work: extracting hidden-state activations from a large sparse mixture-of-experts model to train a small (~500M parameter) custom draft model against a code-and-reasoning dataset. The goal is a purpose-built speculative decoder for one of the fleet's larger models, to be deployed back onto the production rig once validated — an in-progress research effort, not yet a shipped result.