71–80 of 236 posts
Pay attention to IO bandwidth if you’re building a machine with multiple GPUs like this! In this setup the model is sharded between cards so data must be shuffled through a PCIe 3.0 x16 link which is limited to ~16 GB/s max. For reference that’s an order of magnitude lower than the ~350 GB/s memory bandwidth of the Tesla P40 cards being used. Author didn’t mention NVLink so I’m presuming it wasn’t used, but I believe…
Pay attention to IO bandwidth if you’re building a machine with multiple GPUs like this! In this setup the model is sharded between cards so data must be shuffled through a PCIe 3.0 x16 link which is limited to ~16 GB/s max. For reference that’s an order of magnitude lower than the ~350 GB/s memory bandwidth of the Tesla P40 cards being used. Author didn’t mention NVLink so I’m presuming it wasn’t used, but I believe…
Pay attention to IO bandwidth if you’re building a machine with multiple GPUs like this! In this setup the model is sharded between cards so data must be shuffled through a PCIe 3.0 x16 link which is limited to ~16 GB/s max. For reference that’s an order of magnitude lower than the ~350 GB/s memory bandwidth of the Tesla P40 cards being used. Author didn’t mention NVLink so I’m presuming it wasn’t used, but I believe…
Earlier quoted context omitted.
Runs is an overstatement though. With 4 tokens/second you can't use it on production.
Isn’t 4 tps good enough for local use by a single user, which is the point of a personal AI computer?
I doubt it is that efficient. Even though it has 48GB of VRAM, it's more than twice slower than a single 3090 GPU. In my budget AI setup I use 7840 Ryzen based miniPC with USB4 port and connect 3090 to it via the eGPU adapter (ADT-link UT3G). It costed me about $1000 total and I can easily achieve 35 t/s with qwen2.5-coder-32b using ollama.
Earlier quoted context omitted.
I tried running the 8B model on my 8GB M2 Macbook Air through Ollama and it was awful. It took ages to do anything and the responses were bad at best.
Doesn't 8B need at least 16gb of ram? Otherwise, your swapping I would imagine...
Earlier quoted context omitted.
> expensive to run on an overbuilt machine There's a healthy secondary market for GPUs.
The price goes up dramatically once you go past 12GB though, that’s the problem.
I'm seeing 24GB M40 cards for $200, 24GB K80 cards for $40 on eBay.
Great breakdown!. The "own your own AI" at home is a terrific hobby if you like to tinker, but you are going to spend a ton of time and money on hardware that will be underutilized most of the time. If you want to go nuts check out Mitko Vasilev's dream machine. It makes no sense if you don't have a very clear use case that only requires small models or really slow token generation speeds. If the goal however is not…