51–60 of 125 posts
Wouldn’t a cluster of M4 minis cost less and provide more VRAM? There are posts about people getting decent performance for a lot less than 12k USD.
HN loves it some Apple
I would be much more intrested in a piece on what you can train with this kind of rig, rather than the rig itself
Here is some additional journey apart from the rig. https://sabareesh.com/posts/llm-intro/
Earlier quoted context omitted.
Yes! His power supplies are 2x1500 Watt. That puts it at 3KW max which is more than a 20A circuit can provide (2400W). The standard outlet is typically rated at 15 amps or 1800W. And the 15A breaker is on one circuit. You can get 20A circuits but they need to be wired for it, and replacing the breaker won't cut it. Assuming his GPU is ~450W (his number) and power supplies are 80% efficient, well that means he's pulli…
In the US. The UK or EU will do you 3000W out of a standard domestic socket.
It's over current that causes fires.
Earlier quoted context omitted.
The GPU rental market is fairly reasonable. There's lots of companies doing it. (I work at one of them). 4x 4090 can be fetched for around $0.40/hour on some platforms ... about $1.20 on others depending on how available you want it. Regardless, all in, you can do an average 10-or-so-day train for If you want on-prem, wait a few months. The supply of 5000 series (probably announced at CES in a few days) should push m…
don't you need nvlink? feel like an 80gb a100 would start being worth it at a $1.20/4x 4090 price point
Earlier quoted context omitted.
The last time I checked, a modern Threadripper build is a bit over $10,000. So if you have the budget for that but need something GPU-oriented instead, then I could see that being a reasonable option.
The thing is you need a threadripper-class build to make use of 4 GPUs in the first place. Ordinary PCs don't have the PCIe lanes necessary for that. But pricing is okay-ish, have a look at Geohot's Tinybox for turnkey solutions.
All you need is 4x 4090 GPUs to Train Your Own Model -- and $12000 to buy them
I’m glad to know
All you need is 4x 4090 GPUs to Train Your Own Model -- and $12000 to buy them
The GPU rental market is fairly reasonable. There's lots of companies doing it. (I work at one of them). 4x 4090 can be fetched for around $0.40/hour on some platforms ... about $1.20 on others depending on how available you want it. Regardless, all in, you can do an average 10-or-so-day train for If you want on-prem, wait a few months. The supply of 5000 series (probably announced at CES in a few days) should push m…
Wouldn’t a cluster of M4 minis cost less and provide more VRAM? There are posts about people getting decent performance for a lot less than 12k USD.
Why not 3090s? Same VRAM and cheaper. With both setups you'd be limited to 1B. By contrast, you can run 4-bit quants of Llama 70B on two {3,4}090s, and it's still pretty lobotomized by modern standards. You can also train your own model even without GPUs. Just depends on parameter size.