Two-Node HA Homelab Cluster
A self-hosted, highly-available two-node virtualization cluster with a dedicated high-speed replication backplane, a private AI/agent application stack, full observability, and an edge quorum witness — built and operated the same way production infrastructure is.
Project Brief
- Role
- Solo infrastructure owner and operator
- Scope
- A two-node HA virtualization cluster with dedicated replication networking, a private AI application stack, unified observability, and an independent edge quorum witness
- Timeline
- Built and operated in 2026
- Result
- A production-pattern highly-available homelab cluster with automated replication, live migration, unified monitoring, and zero-touch certificate management
Evidence Included
Measured backplane throughput
The dedicated replication link was independently verified at 2.48 Gbps wire speed with zero retransmits under sustained load testing, not assumed from link-speed specs alone.
Genuine quorum resilience
The cluster tolerates a single-node failure without operator intervention, arbitrated by a third, independent witness device rather than a coin-flip or manual failover decision.
Full-stack observability
Compute, containers, and edge network health are all visible in one place, including a custom exporter for hardware (cellular modem) telemetry the platform doesn't expose natively.
2.48 Gbps (verified, 0 retransmits)
Replication link throughput
3-way (2 nodes + external witness)
Quorum model
Zero (live migration)
Migration downtime
The Problem
A single-node home virtualization host is a single point of failure: any hardware fault or maintenance window means downtime for every service it hosts, and there was no live replication, no quorum mechanism to prevent split-brain, and no unified observability across compute, containers, and edge connectivity. The goal was a small but genuinely highly-available cluster, not just two independent boxes that happen to sit next to each other.
My Approach
High-Speed Storage Backplane & Live Replication
- Built a dedicated point-to-point 2.5GbE cross-connect between the two virtualization nodes, isolated from the main network switch, running jumbo frames (MTU 9000) and verified at 2.48 Gbps wire speed with zero retransmits under iperf3 load testing.
- Backed that link with automated, continuous filesystem-level snapshot replication on a 5-minute interval, plus live, zero-downtime VM/container migration between nodes over the same dedicated link.
- Integrated the pair into the hypervisor's built-in HA manager with 3-way quorum tie-breaking via a lightweight external witness device, so a two-node cluster — normally vulnerable to split-brain with only two votes — has a deciding third vote without a third full node.
Private AI & Agent Application Stack
- Deployed a self-hosted AI workspace and agent backend on a fast, separately-replicated storage pool: an LLM chat UI, a private metasearch engine, a vector database, a relational database with vector-search extensions, a local text-to-speech service, and a cache layer — all reachable only from inside the private network, fronted by authenticated HTTPS with automated wildcard certificate issuance.
- Passed through the host's integrated GPU hardware video encode/decode capability into the container running the workspace, rather than relying on CPU-only inference for any media-adjacent workload.
Observability & Secrets Management
- Stood up a unified telemetry stack — a metrics collector, a dashboarding layer, and an alerting engine — monitoring both cluster nodes, their hosted containers, and an edge cellular failover connection's own radio-level telemetry (signal strength, band, tower information) via a small custom exporter written against the modem's own management API.
- Self-hosted a password-manager-compatible secrets vault as the credential store for the rest of the stack, rather than relying on a third-party cloud vault for homelab secrets.
Edge Defense, Ingress & Quorum Witness
- Repurposed a small, low-power single-board computer as the cluster's external quorum witness, an edge reverse-proxy/SSL terminator, a recursive DNS resolver, and an intrusion-prevention bouncer — deliberately keeping the split-brain tie-breaker physically and logically separate from either virtualization node it's arbitrating between.
- Automated wildcard TLS certificate issuance via DNS-01 challenge against the DNS provider's API, so every internally-hosted service gets a valid certificate without manual renewal.
- Layered a recursive, ad-filtering, DNSSEC-validating resolver with an intrusion-prevention system doing dynamic IP-reputation blocking, plus a mesh VPN subnet router for secure remote access without exposing the cluster directly to the internet.
Key Engineering Challenges Solved
Two-node split-brain risk
A two-node cluster has an even vote count, so a network partition between the nodes can't be resolved by majority vote alone. Solved with a third, external quorum witness on independent hardware rather than accepting the split-brain risk or adding a full third virtualization node.
Replication bandwidth vs. the main network
Continuous snapshot replication and live migration traffic would compete with normal network traffic on a shared switch. Solved with a dedicated physical point-to-point link between the two nodes, verified independently for throughput and reliability before relying on it.
Stack
Virtualization & Storage
AI / Application Stack
Observability & Secrets
Edge & Security
Skills Demonstrated
- ▸Highly-available virtualization cluster design, including quorum/split-brain mitigation on small clusters
- ▸Dedicated storage/replication networking separate from general-purpose traffic
- ▸Self-hosted AI application stack deployment with GPU passthrough
- ▸Unified observability across compute, containers, and edge network hardware, including custom exporters for non-standard telemetry sources
- ▸Edge security architecture: automated certificate management, recursive DNS, intrusion prevention, and VPN-based remote access without direct internet exposure