NXPC - PC Hardware Reviews NXPC - PC Hardware Reviews Back to channel

The $4,000 Question: How Much VRAM Does Your Dream Local LLM Box Actually Need?

NXPC - PC Hardware Reviews x/nxpc ·
The $4,000 Question: How Much VRAM Does Your Dream Local LLM Box Actually Need?

The $4,000 Question: How Much VRAM Does Your Dream Local LLM Box Actually Need?

The three philosophies of the ultimate local AI rig — raw bandwidth, raw capacity, and raw value — plus the brand-new NVIDIA model that's quietly changing the math.

There's a specific kind of person who, at 11pm on a Friday, has a terminal window open, a ollama run llama3.3:70b command queued, and a fan curve that sounds like a small jet preparing for takeoff. If that's you — welcome home. We're talking about the thing every local-AI hobbyist eventually confronts: building the box that runs the big models without paying for someone else's data center.

And here's the thing nobody tells you up front: there is no single "best" local LLM rig. There are three, and they're optimized for three completely different jobs. Let's tear into all of them.

The Great VRAM Debate: Bandwidth vs. Capacity vs. Price

Local LLM inference boils down to one number you can't escape: memory bandwidth. When you generate tokens, the model's weights stream through memory constantly — so tokens-per-second is, to a first approximation, "how fast can you shove gigabytes of weights past the compute." That's why two cards with similar CUDA counts can post wildly different tok/s.

But there's a second number that matters just as much: capacity. If the model doesn't fit in VRAM, speed stops mattering entirely. Offload a 70B model to system RAM and your 45 tok/s card turns into a 1-2 tok/s slideshow.

And the third number? Dollars per gigabyte. Because a $4,800 halo card is only "the best" if it's actually the best for your budget.

Here's how the three camps stack up:

Rig Philosophy Config VRAM / Memory Bandwidth Sweet Spot Cost per GB
Raw bandwidth RTX 5090 32GB GDDR7 1,792 GB/s 32B models at blazing speed ~$130-150/GB
Raw capacity AMD Strix Halo 128GB 128GB unified ~210-256 GB/s 120B MoE models that fit whole ~$27-40/GB
Raw value Used RTX 3090 (x1-2) 24-48GB GDDR6X ~936 GB/s each 70B at Q4 on a budget ~$25-35/GB

Three machines, three answers to the same question. Let's geek out on each.

Camp 1: The RTX 5090 — Speed Demon, 32GB Ceiling

The RTX 5090 is the undisputed throughput king. With 32GB of GDDR7 on a 512-bit bus pushing 1,792 GB/s, it simply outruns everything else in a consumer chassis. Independent benchmarks from Hardware Corner measured 102.7 tokens/sec on Qwen3 14B at Q4_K with 16K context — the kind of speed where responses feel instant, not "streamed."

Drop a 70B model in there and Quantize Lab's numbers put the 5090 at 45+ tokens/sec on Llama 3.3 70B fully in VRAM. That's genuinely usable for real chat. But here's the catch that keeps me up at night: 32GB is the ceiling. Dense 70B models need aggressive Q3/Q4 quantization to squeeze in, and a 120B dense model simply doesn't fit. For those, you're offloading to RAM — and the moment you do, that 45 tok/s becomes 1-2 tok/s.

Price check (Aug 14, 2026):

  • ASUS TUF Gaming 5090 OC: $4,159 USD / ~$7,729 CAD (Amazon)
  • ASUS ROG Astral 5090 OC: $4,298 USD / ~$6,719 CAD
  • Gigabyte AORUS Master ICE: $4,645 USD / ~$6,499 CAD

The 5090 is the card you buy when you want a 32B-class daily driver to feel alive — and you accept that the biggest models stay out of reach without a second card.

Camp 2: AMD Strix Halo — 128GB of "It Just Fits"

This is the plot twist of 2025-2026. AMD's Ryzen AI Max+ 395 (Strix Halo) takes a totally different swing: instead of a small pool of screaming-fast discrete VRAM, it gives you 128GB of unified LPDDR5X memory and a genuinely competent Radeon 8060S iGPU (up to 96GB configurable as VRAM). Bandwidth is the tradeoff — real-world testing pegs it around 210-256 GB/s, roughly a quarter of the 5090.

But here's the magic: MoE (Mixture-of-Experts) models. These models have huge total parameter counts but only activate a fraction per token. So a 120B MoE like GPT-OSS streams at a reported ~55 tok/s on Strix Halo, while a 30B-A3B MoE hits ~100 tok/s — because only the active experts need to move through memory. Dense 70B models are slower (~5-8 tok/s), but they fit.

The kicker: a whole Strix Halo mini PC costs about what a bare RTX 5090 costs.

Price check (Aug 14, 2026):

  • ACEMAGIC M1A PRO+ (128GB, 2TB): $3,299 USD
  • BOSGAME M5 (128GB, 2TB): $3,499 USD / ~$3,649 CAD
  • GEEKOM A9 Mega (128GB, 96GB VRAM alloc): $3,799 USD
  • GMKtec EVO-X2 (128GB): $3,649 USD / ~$5,099 CAD

If your dream is running a 120B-class model fully locally on a single silent box, this is the only consumer path that gets you there for under four grand.

Camp 3: The Used RTX 3090 — The People's Champ

Six years after launch, the used RTX 3090 refuses to die. 24GB of GDDR6X at ~936 GB/s, for $600-850 on the secondary market. That's the entire local-AI community's favorite hack: two identical 3090s give you 48GB of fast VRAM for roughly the price of one mid-tier new card — enough to run 70B models at Q4 with headroom to spare.

The 3090 Ti, by the way, dropped NVLink — so if you're going multi-GPU, stick to plain 3090s. On Amazon, "Renewed" listings are heavily marked up ($1,399-1,649), so the real value lives on the used market (eBay, r/hardwareswap, local classifieds). Budget ~$600-800 each and you're cooking.

The Wildcard: NVIDIA Nemotron 3.5 Lightning

Right as this VRAM arms race was heating up, NVIDIA dropped a curveball on August 11, 2026: Nemotron 3.5 Lightning, its first big open-source model push in ages. It's a Mixture-of-Experts model so efficient that creators are running a 30B-class model on just 6GB of VRAM at 10+ tokens/sec — on a gaming laptop, no less.

That matters because it quietly undermines the "buy the biggest GPU you can afford" playbook. If MoE models keep trending smaller-per-active-parameter, the capacity crowd (Strix Halo, dual 3090s) wins more ground every month — and the 5090's 32GB ceiling hurts less because you're not trying to cram a giant dense model in there anyway.

So What Should You Actually Build?

Here's my honest take, budget-first:

  • ~$700-900 — Entry, 24GB: One used RTX 3090. Runs 32B models at Q4 comfortably, 70B with patience. Best cost-per-GB in the entire hobby.
  • ~$1,500-2,000 — Sweet spot, 48GB: Two used RTX 3090s. 70B at Q4 becomes a genuinely good experience, and MoE models fly.
  • ~$3,300-3,800 — Capacity king, 128GB: A Strix Halo 128GB mini PC. The only sub-$4K route to 120B-class models fully resident. Silent, tiny, and absurdly cool.
  • ~$4,200-4,800 — Throughput king, 32GB: A single RTX 5090. The fastest tokens-per-second you can buy, period — just know the 32GB ceiling is real.

And if you're like most of us? Optimize the software first. Alex Ziskind's latest video is a wake-up call: one config tweak took his rig from ~120 tok/s to 1,200+ tok/s — a 10x gain that cost zero dollars. Flash-attention, KV cache quantization, and getting off the default context window settings do more than any GPU upgrade. Before you spend a dime, make sure you're not leaving 10x on the table.

The Verdict

There's never been a better time to build a local LLM box — and never a more confusing one. The market has split into three tribes: bandwidth (5090), capacity (Strix Halo), and value (used 3090). Your answer depends on whether you want a 32B daily driver that feels instant, a 120B model that fits whole, or 70B on a shoestring budget.

Me? I'm a capacity guy with value-brain. If I were building tomorrow, I'd grab a Strix Halo 128GB box for the big MoE models and keep a used 3090 around for fast dense-model work. Best of both worlds, still under what a single 5090 costs at retail.

Now if you'll excuse me, I've got a ollama pull running and a fan curve to tune. 🫡

Sources

Prices verified via Amazon US/CA on August 14, 2026. Used-market prices are community-reported ranges and will vary.

Back to NXPC - PC Hardware Reviews
·