Local LLM Inference Server
A self-hosted, dual-GPU inference rig serving a fleet of large language models with hot-swapping, speculative decoding, full observability, and rigorously validated performance tuning.
Project Brief
- Role
- Solo infrastructure owner and operator
- Scope
- A dedicated dual-GPU Linux server exposing a growing OpenAI-compatible inference fleet (10 models and counting) to day-to-day developer tools, plus external agent tooling over a public gateway
- Timeline
- Continuously operated and tuned in 2026
- Result
- A production-like local AI platform with live telemetry, repeatable performance validation, and up to 256K-token context
Evidence Included
Live production telemetry
When the rig is awake, the panel below reports the real model, GPU load, VRAM, temperature, throughput, and token activity.
Measured tuning results
The case study records benchmarked MTP speedups, draft acceptance, memory-clock gains, and sustained stress validation.
Public implementation context
The architecture, model-swapping behavior, capacity constraints, and observability stack are documented in detail on this page.
Live debugging, not just a demo
Two real bugs in the status service itself — a stale model cache and a biased GPU-utilization reading — were root-caused and fixed with this site's own live feed as the test harness.
2x 16GB VRAM
GPUs
10, 9B–35B
Model fleet
256K tokens
Max context
~1.5–2x (MTP)
Decode speedup
65–80%
Draft acceptance
5–30 sec
Cold model swap
~17% effective clock
Memory OC gain
6x sustained runs
Stress validation
ant@antllm:~$ status --watch
connecting
raw feed ↗ (opens in new tab)establishing uplink…
A home-built local AI infrastructure project: a dedicated Linux server running a dual-GPU (2x 16GB VRAM) inference rig behind an OpenAI-compatible API. Rather than keeping one model loaded at all times, the stack hot-swaps between a growing fleet of open-weight LLMs (9B–35B parameter class) on demand, automatically freeing and reallocating GPU memory per request — all behind one stable endpoint any OpenAI-compatible client or coding agent can talk to. Every performance change (GPU clocks, decoding strategy, context sizing) was validated with real before/after benchmarks rather than assumed — the same rigor expected of production infrastructure work. The rig is also self-monitoring: a small read-only, key-authed status API feeds the live panel on this page, and has itself been debugged and hardened in production rather than just built once and left alone — see the highlights below for two real bugs found and fixed in it. It's since grown from an internal-only rig into a public, authenticated AI provider — reachable by external coding agents and OpenAI-compatible tools, not just clients on the home network.
Highlights
- Dual-GPU inference server running a model-swapping proxy (llama-swap) in front of llama.cpp — requesting a different model automatically unloads the current one and loads the next, cold in 5–30 seconds and instant on every request after that, with zero manual intervention
- Up to 256K-token context across the entire model fleet, including multimodal (vision) support on one model
- Self-speculative multi-token-prediction (MTP) decoding on 5 of the fleet's 10 models, cutting generation time by ~1.5–2x at a 65–80% draft-acceptance rate — no separate draft model to manage
- Extended the rig beyond the LAN into a public, authenticated AI provider — external coding agents and OpenAI-compatible tools can now reach the fleet from anywhere over an outbound-only tunnel, with zero inbound firewall changes; the raw inference port itself stays exactly as LAN/VPN-only as before
- Fronted that public route with a purpose-built gateway enforcing four independent layers before a request ever reaches a model — bearer-key authentication, an endpoint allowlist (only chat/completion-style routes resolve; admin, metrics, and UI endpoints 404 regardless of auth), a concurrency guard matched to the fleet's real one-request-at-a-time GPU capacity so overlapping requests queue briefly instead of colliding, and per-key rate limiting — plus a second, independently-enforced rate limit at the network edge as defense in depth
- Verified the gateway end-to-end from the public internet before trusting it with real traffic — rejected invalid/missing auth, confirmed admin-style routes stay unreachable even with a valid key, exercised a full real chat-completion round trip, and proved the concurrency queue behaves correctly under overlapping requests rather than erroring
- Full observability stack — Prometheus feeding three Grafana dashboards (one custom-built plus two community-standard imports for host and GPU metrics) tracking utilization, VRAM, temperature, and power in real time
- Validated GPU memory overclocking through structured A/B testing — ~17% higher effective memory clock with zero throughput regression across 6 sustained stress cycles and a full reboot re-validation
- Caught a live GPU core-clock change silently collapsing one model's MTP draft acceptance from 65–80% to 0% (and decode speed from 73 to 29 tok/s) — root-caused, reverted, and turned into a standing rule: clock changes only happen at boot, never against a running inference session
- Benchmarked a new coding-focused model (Ornith 1.0 35B) against the fleet leader on public benchmarks before adopting it — up 2.2 pts on SWE-bench Verified, 11.7 on Terminal-Bench 2.1, and 5.2 on NL2Repo — and kept it as an A/B candidate pending real-workload validation rather than an outright replacement
- Self-monitoring by design — a key-authed, rate-limited, CORS-locked status API feeds the live panel on this page over a Tailscale Funnel, with every other service on the box firewalled to LAN/VPN-only
- Found and fixed two real bugs in that same status service: a stale in-memory model cache that hid newly added models until a manual restart, and a GPU-utilization reading stuck near 0% because it sampled a bursty, sub-second signal once every 20 seconds — root-caused by polling nvidia-smi directly during a live generation, fixed with config-driven capability detection and sub-sampled averaging, and verified live end-to-end both times
- Replaced the panel's single blended cloud-cost-avoided rate with an itemized input/output/cached model priced against the single most expensive real hosted quote for a comparable model — a deliberate conservative upper bound, not an average — including reconstructing a lifetime KV-cache-reuse token counter from llama.cpp's own /slots endpoint, verified live against the API's own reported cache-hit figures, since this build's /metrics didn't yet expose one as a cumulative counter
- Extended MTP decoding to a text-only profile of the vision-capable model, and — while chasing an unrelated GPU-overclock question — isolated and documented a reproducible non-determinism quirk specific to that model's vision-encoder path, ruling out hardware instability through controlled, repeatable testing
- Wired into day-to-day developer tooling — coding agents and IDE integrations route requests to the local fleet exactly like a cloud LLM provider
Architecture & Infrastructure
Model-swapping API gateway
A lightweight proxy (llama-swap) sits in front of llama.cpp and exposes one stable OpenAI-compatible endpoint. Requesting a different model by name transparently unloads the current one, frees its VRAM, and loads the next on its own internal port — clients never need to know which model is physically resident. An idle model is also auto-unloaded after an hour to clear accumulated host RAM/context-cache buildup, without anyone having to remember to do it.
VRAM-aware capacity planning
With 32GB of VRAM total across two GPUs, only one 25–35B-class model fits at a time. Every model in the fleet is tuned (quantization level, KV-cache precision, context length) to leave a safe, measured VRAM margin at full 256K context rather than guessing at headroom.
Speculative decoding for real throughput gains
3 of the fleet's models use self-speculative multi-token prediction for faster decoding. Most bake the draft head directly into the model weights; one instead pairs the base model with a small, separate official drafter file — the two MTP implementations llama.cpp currently supports. Either way, the draft proposes several tokens ahead and the main model verifies them in a single pass, translating a measured 65–80% draft-acceptance rate into a genuine ~1.5–2x faster generation.
Full-stack observability
Prometheus scrapes GPU, host, and inference-server metrics into three Grafana dashboards: a hand-built one tailored to this box, plus two widely-used community imports (Node Exporter Full and NVIDIA GPU Metrics) adopted instead of reinventing well-covered ground — the same build-vs-adopt judgment call expected on a production observability stack.
Evidence-based performance tuning
GPU memory overclocking was pushed and validated incrementally with repeatable benchmarks (determinism checks, sustained stress runs, full reboot cycles) rather than applied blindly, and an unstable setting was rolled back before it could affect real use. Separately, a live core-clock change — applied while inference processes already held an active CUDA context — silently collapsed one model's MTP draft acceptance from 65–80% to 0% and its decode speed from 73 to 29 tok/s; reverting the clock value alone didn't fix it, only a full service restart did. Root-caused and turned into a standing rule: clock changes only ever happen at boot, never against a running inference session.
Benchmark-driven model evaluation
New models are evaluated against the current fleet leader on public, third-party benchmarks (SWE-bench Verified, Terminal-Bench, NL2Repo) before being promoted — Ornith 1.0 35B beat the incumbent on all three, but ships as a coexisting A/B candidate rather than an outright replacement until it's proven out on real workloads, not just benchmark scores.
Self-monitoring, hardened public telemetry
A small, dependency-free status API on the box itself tracks lifetime token counts, per-model best-observed throughput, and rolling activity history, persisted across model swaps and restarts. It's the only service on the box reachable from the public internet — key-authed, origin-locked, rate-limited to 30 requests/10s, and funneled out over Tailscale rather than a forwarded router port — with every other service on the host firewalled to LAN/VPN-only, and the API key already rotated once as a precaution.
Public AI provider gateway
A purpose-built gateway fronts a public, outbound-only tunnel into the fleet so external coding agents and OpenAI-compatible tools can use it too, layering four independent controls in front of every request: bearer-key authentication, an endpoint allowlist (only chat/completion routes resolve — admin, metrics, and UI endpoints 404 regardless of auth), a concurrency guard sized to the fleet's real one-request-at-a-time GPU capacity so overlapping calls queue briefly rather than collide, and per-key rate limiting, backed by a second, independently-enforced rate limit at the network edge. The underlying inference port itself never becomes internet-facing — every other service on the box stays LAN/VPN-only exactly as before. Verified end-to-end from the public internet — auth rejection, admin-route lockout, a real completion round trip, and correct queuing under overlapping requests — before it carried any real traffic.
Operational debugging, not just uptime
The status service itself has needed real fixes, found and resolved the way any production issue would be. A model roster fetched once at process start and cached forever (so a newly added model stayed invisible until a manual restart) was replaced with a live refresh every poll cycle; a hardcoded vision/MTP allowlist that could silently drift from the real config was replaced with a scan of the config's own launch flags; and a GPU-utilization reading stuck near 0% almost permanently was traced — by polling nvidia-smi directly during a live generation — to a single instantaneous sample of a signal that only spikes for milliseconds at a time, then fixed by sub-sampling and averaging across each reporting window. Every fix was verified against the live public endpoint before and after, with no downtime.
Itemized cost modeling instead of one blended rate
The original “cloud cost avoided” estimate priced every token — prompt and generated alike — at one flat rate, and quietly excluded prompt tokens from the total entirely. Real hosted-inference vendors bill input, output, and cached-input tokens separately, and this fleet spans a wide compute range (small-active-parameter MoE models alongside dense 27–35B reasoning models) — averaging across that spread would let the cheap end quietly understate the total, so the panel instead prices against the single most expensive real hosted quote found for a comparable model (DeepSeek R1 Distill Qwen 32B, surveyed across 6 providers) as a deliberately conservative upper bound rather than a best-case average, plus a ~90% cache discount on input — the documented industry-standard rate OpenAI, Anthropic, and Gemini all use for prompt caching. It's paired with a genuine lifetime cache-hit counter reconstructed from llama.cpp's /slots endpoint (tracking each completed request's real KV-cache-reuse count, verified live against the API's own reported cache-hit figure, since this build's /metrics doesn't expose a cumulative counter for it) rather than assumed away as zero.