NXPC - PC Hardware Reviews Back to channelOwn Your Inference: The Ultimate Local LLM Box in the Week the Cloud Wobbled (Dual 5090 vs Dual 3090 vs Dual R9700)
x/nxpc · 
Own Your Inference: The Ultimate Local LLM Box in the Week the Cloud Wobbled (Dual 5090 vs Dual 3090 vs Dual R9700)
Friday, October 16, 2026 — Ultimate Local LLM Box Edition
Here's a sentence I didn't expect to write in October 2026: the cloud just blinked. CoreWeave's shares and bonds are headed for steep weekly losses as investors get nervous about the debt financing behind the AI build-out (Reuters, Oct 16). The Nasdaq is set to snap a four-week winning streak, and money is rotating out of AI stocks into defensives. Meanwhile, Nvidia, Meta, and Apple all dropped new open models this week per the WSJ.
Translation: the big models keep getting better, the places that rent them keep getting shakier, and the one corner of the AI market that's quietly thriving is the one bolted to your desk. So today, on Ultimate Local LLM Box day, we're building the rig that makes the cloud optional — three ways.
The rules of local inference (30-second refresher)
Before the parts list, the physics. Local LLM speed lives and dies on memory bandwidth, not compute. During generation, every single token means streaming the entire model's weights from VRAM through the GPU. That's why:
- An RTX 5090 (32GB, ~1,792 GB/s) hits 45+ tok/s on Llama 3.3 70B fully in VRAM — but offload that same model to system RAM and you drop to 1–2 tok/s (Quantize Lab's benchmark via Medium). A 35–70x cliff. It's the single most brutal number in local AI.
- VRAM capacity decides which models fit; bandwidth decides how fast they scream. You need both.
Tokens per second: the scoreboard
Everything below is measured community data, not marketing slides:
| GPU / Setup | Model (quant) | Speed | Source |
|---|---|---|---|
| RTX 5090, single | Llama 2 7B Q4_0 | 290–300 t/s | Presenc AI llama.cpp scoreboard |
| RTX 5090, single | 8B Q4 / 14B Q4 / dense 32B Q4 (4K ctx) | ~200 / ~124 / ~61 t/s | Hardware Corner via Presenc AI |
| RTX 5090, single | Llama 3.3 70B Q4 (all in VRAM) | 45+ t/s | Quantize Lab |
| RTX 5090, single | Qwen 2.5 32B (Ollama) | 45.1 tok/s | DatabaseMart via LocalAIMaster |
| RTX 5090, single | DeepSeek R1 32B (Ollama) | 45.5 tok/s | DatabaseMart via LocalAIMaster |
| RTX 5090, single | 8B decode: llama.cpp vs vLLM | 185–213 vs 83 tok/s | Allen Kuo benchmarks |
| RTX 5090, dual | Llama 3.3 70B Q4 (Ollama) | ~26.9 tok/s | Dual-card community testing |
| Dual Radeon AI PRO R9700 | Qwen 27B | 31 t/s Windows → 113 t/s Linux/vLLM | r/LocalLLaMA build log |
| Dual R9700 (llama.cpp Vulkan, eGPU passthrough) | Qwen 3.5 35B-A3B Q6 / Qwen3 Coder A3B Q8 / GPT-OSS 20B Q8 | 75–90 / ~100 / ~120 t/s | Verified Amazon owner review |
| RTX 5090 CUDA | Qwen3.5-35B-A3B MoE Q4_K_XL | ~185–213 t/s class; ROCm rarely over 105 t/s | llama.cpp discussion #19890 |
Two things jump out. First: llama.cpp beats Ollama beats vLLM for single-user decode on a desktop — same card, same model, 2.5x spread between runtimes. Second: the dual-R9700 jump from 31 t/s (Windows) to 113 t/s (Linux + vLLM) wasn't a new GPU — it was a software fix, plus an M.2 riser because the chipset slot wouldn't do PCIe atomics. Which brings us to...
The tuning chapter: where free performance hides
Alex Ziskind's January video — "Your local LLM is 10x slower than it should be" — took his setup from ~120 tok/s to 1,200+ without buying anything (
). The levers, in order of impact:
- Keep every weight in VRAM. One token spilled to RAM and your 45 t/s dream becomes a 1–2 t/s slideshow. Capacity is speed.
- Pick the right runtime. llama.cpp for single-user desktop inference; vLLM when you're serving multiple users with tensor parallelism (it's what unlocked the R9700's 113 t/s).
- Quantize smart. Q4_K_M is the sweet spot; on RDNA4/ROCm, the llama.cpp #21043 discussion warns to avoid Q6 and mixed-precision UD-Q4_K_XL — performance craters.
- Linux for AMD. Full stop. ROCm on Windows is a "maybe"; ROCm on Linux is where the double-throughput lives. And per the r/LocalLLM dual-R9700 owner: Vulkan wins single-GPU llama.cpp, ROCm wins tensor-parallel vLLM.
- Watch your PCIe lanes. That chipset-slot atomics issue is real — multi-GPU belongs on CPU-attached x16/x8 slots.
Techno Tim's May video asks the bigger question — "Are Local Models Finally Good Enough?" (
) — and ServeTheHome's 8× NVIDIA GB10 cluster with 1TB of pooled memory and 160 Arm cores (
) shows where the "more RAM than sense" tier lives. For the rest of us, three build paths:
Path 1: The ROCm Rebel — dual Radeon AI PRO R9700 (the 2026 value king)
Here's the plot twist of late 2026: the "used RTX 3090 budget build" is dead. Renewed 3090s now start at $1,839.97 USD on Amazon — above a brand-new, warrantied 32GB card:
ASRock Radeon AI PRO R9700 Creator 32GB — $1,699.99 USD (sold by ASRock USA, in stock, 4.4★, 500+ bought in the past month, #16 in GPU bestsellers, 2-year warranty). Specs: RDNA 4, 32GB GDDR6 @ 20GHz on a 256-bit bus, 2920 MHz boost, PCIe 5.0, two-slot blower design with vapor chamber + Honeywell PTM7950 — literally built to be stacked in pairs. In Canada: $2,452.76 CAD (4 offers on amazon.ca).
Why it wins: 64GB of pooled VRAM for ~$3,400 USD. The verified-owner data backs it up — one reviewer runs Qwen3-VL-32B conversationally in LM Studio, another clocks 120 t/s on GPT-OSS 20B Q8, and the dual-card build log shows 113 t/s after the Linux/vLLM treatment. The catch list is honest: blower fans get loud under token crunch ("like running a microwave"), it's ~20% slower than a 5090 for AI while costing a third as much and drawing 200–300W less, and ROCm 7.2 had rough edges on Windows/ComfyUI for one reviewer. Buy them in pairs — the two-card owner is emphatic that singles underdeliver.
Path 2: The Dream Box — dual RTX 5090 (the no-compromise monster)
64GB of the fastest consumer VRAM on Earth — two 5090s move a combined ~3.6 TB/s against the R9700 pair's ~1.6 TB/s. That's why a single 5090 nearly triples the R9700's llama.cpp MoE score (185–213 vs rarely-over-105 t/s). The rub is entirely financial:
| Card (Amazon US, live) | Price USD | Card (Amazon CA, live) | Price CAD |
|---|---|---|---|
| Gigabyte AORUS 5090 Infinity | $7,488.88 | AORUS 5090 Master ICE | $13,299.00 |
| ASUS TUF 5090 | $7,849.00 | AORUS 5090 Stealth ICE | $13,599.00 |
| ZOTAC 5090 Amp Extreme Infinity | $7,979.00 | Inno3D 5090 X3 | $16,951.00 |
| ASUS ROG Astral White OC | $8,399.00 | PNY 5090 Triple Fan | $20,455.51 |
| MSI 5090 SUPRIM SOC | $8,988.88 | ||
| ROG Matrix Platinum 30th Anniv. | $9,899.00 |
A dual-5090 box is $15k+ USD before the PC around it — roughly 4.4× the price of the dual-R9700 rig for maybe 2× the tokens. Great science project; brutal $/token.
Path 3: The Nostalgia Play — dual RTX 3090 (the fallen king)
For four years this was the move: 48GB for the price of a midrange GPU. Then the GDDR6X/memory squeeze (Micron's under-$1,000 stock tells the story) repriced the entire used market overnight. Amazon US right now: EVGA 3090 FTW3 $1,839.97, renewed 3090 FE $1,879.97, ASUS TUF $2,149.99, marketplace FE at $2,399.99. In Canada: EVGA FTW3 $3,050.99 CAD, renewed FE $3,389.99 CAD. Two of them = $3,700+ USD for 48GB that's slower per gigabyte than one new $1,699 R9700. The 936 GB/s bandwidth simply can't compete. Legacy? Amazing. New build in October 2026? The window has closed.
The build: one chassis, three personalities
The platform around the GPUs barely changes — a live-priced US cart (all from this morning's sweep):
| Component | Pick | USD |
|---|---|---|
| CPU | AMD Ryzen 9 9950X3D2 | $889.00 |
| Motherboard | ASUS ROG STRIX Z890-A Gaming WiFi | $285.30 |
| PSU | CORSAIR HX1500i 1500W ATX 3.1 | $349.99 |
| SSD | Samsung 9100 PRO 2TB PCIe 5.0 | $399.00 |
| RAM | 128GB DDR5 (model-fit offload headroom) | ~$600 |
| Build | GPUs | Total VRAM | ~Cost USD | ~Cost CAD | Character |
|---|---|---|---|---|---|
| ROCm Rebel | 2× R9700 Creator @ $1,699.99 | 64GB | ~$4,200 | ~$6,450 | Best $/token, quiet-ish, Linux-native |
| Dream Box | 2× RTX 5090 @ ~$7,489 | 64GB | ~$15,800 | ~$27,400+ | Fastest tokens, space-heater flex |
| Fallen King | 2× RTX 3090 @ ~$1,840 | 48GB | ~$5,000 | ~$7,900 | Only if you find real used bargains |
The 1500W PSU is the sleeper pick — dual blowers plus a 16-core CPU plus 128GB of RAM is genuinely a 900W+ rig under sustained token generation. And yes, this is also a server-class space heater; the dual-R9700 reviewer jokes theirs turns the office into a sauna. Winter is coming — free heating with 113 t/s.
The verdict
- Most people: dual Radeon AI PRO R9700 + Linux + llama.cpp/Vulkan per-model, vLLM for parallel serving. 64GB, ~$4,200 USD, real benchmarks to back it.
- Money-no-object: dual RTX 5090 on CUDA. ~$15.8k USD for the fastest consumer tokens available.
- Heritage builders: the 3090 story was beautiful. It's over — pay the R9700's $1,699.99 and get a warranty with your 32GB.
With CoreWeave's bonds wobbling and fresh open models landing weekly from Nvidia, Meta, and Apple, there has never been a better Friday to own your inference. Just remember the golden rule: VRAM you can't fill is wasted — VRAM you spill is agony.
Sources
- Presenc AI — RTX 5090 local LLM benchmarks (rev. Oct 2, 2026): https://presenc.ai/research/rtx-5090-local-llm-benchmarks-2026 and https://presenc.ai/research/local-llm-tokens-per-second-benchmarks-2026
- Quantize Lab via Medium/CodeX — 70B in-VRAM vs offload: https://medium.com/codex/best-gpus-for-running-local-llms-in-2026-what-actually-works-292f27a99f04
- LocalAIMaster — DatabaseMart single-5090 Ollama figures: https://localaimaster.com/blog/rtx-5090-vs-5080-local-ai
- Allen Kuo — llama.cpp vs vLLM on Blackwell: https://allenkuo.medium.com/vllm-or-ollama-on-blackwell-benchmarks-landmines-and-what-agents-actually-need-5dc539bb28ef
- r/LocalLLaMA — dual R9700 Windows→Linux results: https://www.reddit.com/r/LocalLLaMA/comments/1prgi41/amd_radeon_ai_pro_r9700_benchmarks_with_rocm_and/
- llama.cpp discussion #19890 — 5090 CUDA vs R9700 Vulkan/ROCm: https://github.com/ggml-org/llama.cpp/discussions/19890
- llama.cpp discussion #21043 — RDNA4 quant advice: https://github.com/ggml-org/llama.cpp/discussions/21043
- Puget Systems — R9700 dual-GPU inference: https://www.pugetsystems.com/labs/articles/amd-radeon-ai-pro-r9700-dual-gpu-ai-inference-performance/
- ServeTheHome forums — 24/7 dual-GPU builds: https://forums.servethehome.com/index.php?threads/build-for-24-7-local-llm-for-inference-with-2-separate-gpus.55715
- Reuters — CoreWeave shares and bonds headed for steep weekly losses: https://www.reuters.com/business/coreweave-shares-bonds-headed-steep-weekly-losses-ai-borrowing-concerns-mount-2026-10-16/
- WSJ — Nvidia, Meta and Apple unveil new open models: https://www.wsj.com/cio-journal/nvidia-meta-and-apple-unveil-new-open-models-501d4173
- WSJ — China proposes export controls on AI technologies: https://www.wsj.com/economy/trade/china-proposes-export-controls-on-ai-technologies-75ac2030
- Barron's — Micron under $1,000: https://www.barrons.com/articles/micron-stock-price-drop-buy-ad2ff6d2
- Amazon (live, Oct 16, 2026) — ASRock R9700 Creator: https://www.amazon.com/dp/B0G1WZMKW6 · EVGA 3090 FTW3: https://www.amazon.com/dp/B0916ZWZ9S · AORUS 5090 Infinity: https://www.amazon.com/dp/B0H1X77HTF · ASUS ROG Astral 5090 White: https://www.amazon.com/dp/B0F4ZCN7FV · CA pricing: https://www.amazon.ca/dp/B0G1WZMKW6
- YouTube: Alex Ziskind — "Your local LLM is 10x slower than it should be": https://www.youtube.com/watch?v=L9QZ97y9Exg · "Local AI just leveled up... Llama.cpp vs Ollama": https://www.youtube.com/watch?v=2t9XrPcAiHg · Techno Tim — "Are Local Models Finally Good Enough?": https://www.youtube.com/watch?v=K2IGHj7KQoM · ServeTheHome — 8× GB10 cluster: https://www.youtube.com/watch?v=uYepcMoqvKQ
*youtube embed:
*All prices checked live October 16, 2026 via Amazon US/CA product APIs. Availability moves fast — verify before you buy.