AINews wire, Yesterday

Homelab runs a 2.8-trillion-parameter AI model on two RTX 3090s and Optane memory

Reported by Tom's Hardware

Image: Tom's Hardware

A Reddit homelabber has upgraded a second-generation Optane memory server so it can run Moonshot AI's Kimi K3 model, according to Tom's Hardware. The model has 2.8 trillion parameters. The builder added two RTX 3090 graphics cards to the machine. In the builder's measurements, the system now generates 3.11 tokens per second. Before the GPUs were added, the same build ran at 2.20 tokens per second. The update was shared in a follow-up post on Reddit.

The largest single gain came from moving part of the model into a RAM drive. Of the model's 94 shards, 45 now sit in system memory. That is 414 GB of the 865 GB model file. The rest stays in Optane persistent memory. The builder reports that this move raised generation speed by 41 percent and prompt evaluation by 26 percent. Prompt evaluation now reaches 16.8 tokens per second. A RAM drive is volatile, so its contents are lost when power goes off, although it behaves like a disk. The source notes that not every owner has that much spare RAM.

The two GPUs hold only 48 GB of VRAM. That is roughly 5.5 percent of the model file, so they cannot hold the whole model. The setup works because the model uses a mixture-of-experts design. Only a few expert groups run for each token. The GPUs handle attention, the dense feed-forward layer, the shared experts and the KV cache. The routed experts stay on the CPU side, in Optane and the RAM drive. The model has 896 experts. For each token, 16 are chosen and two are shared, which gives about 104 billion active parameters. The Q2_K file used here is under 3 bits per parameter.

The builder says the speed from the GPUs fell within the range expected from the original build, which was 2 to 4 tokens per second. Cooling remains a serious problem. The builder says the improved throughput does not change the original thermal warning. Two small blower fans for the Optane modules stay in place. Fitting the dual GPUs required cutting metal from the case and moving the power supply. The builder may try custom quantization next, which could reduce pressure on the slower memory tiers. Any such change would also need checks on output quality, because speed alone does not show whether answers are good.

What is still unknown: the source gives no figures on output quality, total build cost or power use. The results come from one builder's own tests, not from an independent lab. Tom's Hardware reports these numbers as measured by the builder, so a different machine may get different results.

Enjoying this? Get the AI brief

Every Thursday: AI that matters for your home.

✓ Free · seven editions · one click to leave

Topics
Story continues on Tom's HardwareAdd us as a preferred source on GoogleSee our stories first in Google’s Top Stories.
Get alerts on big newsOnly the big news, at most once a day. No email, no account.

Original reporting by the sources above. Summary by smart-home.space.