AINews wire, 1 h ago
An old GTX 1080 can run large local AI models at home, with slow replies
Reported by XDA Developers

A reader with an older graphics card may be able to run large AI models at home. In a test reported by XDA Developers, the author used a GTX 1080 with 8GB of VRAM, a card Nvidia no longer sells, to run local large language models. The setup used a Proxmox host with a Debian container. The key was a mixture-of-experts model, which can keep part of its weights in ordinary system memory.
The author said the newest drivers for this card did not work well with Proxmox kernels. The article names two driver versions, 580.119.02 and 580.173.02, and does not explain the difference. The driver was installed on the host first. Then it was added to a Debian container, where GPU passthrough was set up with a community script. The container needed an extra flag so the driver installer would skip its kernel modules. Without that flag, the installation failed.
For the inference engine, the author chose llama.cpp over Ollama. He described llama.cpp as lighter, which left more performance for the model, and as offering more settings for tuning. He used the Vulkan backend rather than CUDA, because Vulkan caused fewer compatibility problems on this older Pascal-era card. The main test model was Qwen3.6-35B-A3B, a 35-billion-parameter model in a quantised file. The context size was set to 100,000 tokens. Most of the expert weights were moved to the CPU and system RAM, while the attention layers and the active experts stayed on the GPU.
The reported speeds were 50 to 70 tokens per second in prefill, the stage where the model reads the prompt. In the second stage, the author measured about 15.2 tokens per second. He called this impressive for a ten-year-old card with no Tensor cores and only 8GB of VRAM. A second model, Gemma-4-26B-A4B, showed similar prefill speed and a slightly lower output rate of 14.6 tokens per second. The author also said lighter models run well on the same card for everyday productivity tasks.
The main group affected is people with older gaming PCs, spare servers or home lab setups who want an AI tool that runs on their own hardware. The author ran other self-hosted services on the same Proxmox host, so the same machine can serve both purposes. Speeds of around 15 tokens per second are workable for some tasks but slower than most cloud chatbots. Results depend on the model, the quantisation, the context size and the amount of system RAM, so readers should not expect the same numbers without testing.
The article is based on one person's setup and does not give power use, noise, or the total cost of the hardware. It does not say how the results would change with other GPUs, other operating systems or other prompts. It also leaves open why two driver versions appear in the steps, and which one a reader should use for a new install.
Enjoying this? Get the AI brief
Every Thursday: AI that matters for your home.
✓ Free · seven editions · one click to leave
Original reporting by the sources above. Summary by smart-home.space.