91–100 of 125 posts
I see a lot of guides but most focus on getting the toolchain up and running, and not much talk about what kind of dataset do I need to do a good fine tuning.
Any pointers appreciated.
Earlier quoted context omitted.
The bottleneck for most model training sizes is VRAM, and since each 4090 has 24 GB VRAM, that's 96 GB VRAM total. The article mentions that it can train LLMs from scratch up to 1 billion hyperparameters, which tracks. Nowadays that's not a lot: a single H100 that you can now rent has 80 GB VRAM, and doesn't have the technical overhead of handling work across GPUs.
Is there a reason you used hyperparameters rather than parameters? I was going to politely correct the terminology but you seem to be in AI for some time so either it was a mistype or I am misunderstanding what you are referencing.
Earlier quoted context omitted.
The bottleneck for most model training sizes is VRAM, and since each 4090 has 24 GB VRAM, that's 96 GB VRAM total. The article mentions that it can train LLMs from scratch up to 1 billion hyperparameters, which tracks. Nowadays that's not a lot: a single H100 that you can now rent has 80 GB VRAM, and doesn't have the technical overhead of handling work across GPUs.
I doubt the RAM is added up. I think that’s only a feature reserved for their NVLinked HPC series cards. In fact, without nvlink, I don’t see how you’d connect them together to compute a single task in a performant and efficient way.
It's a rabbit hole I stay away from for pragmatic reasons.
Earlier quoted context omitted.
In the US. The UK or EU will do you 3000W out of a standard domestic socket.
In the UK they get that by doubling the voltage. The current draw will still be similar to the US. It's over current that causes fires.
Wouldn’t a cluster of M4 minis cost less and provide more VRAM? There are posts about people getting decent performance for a lot less than 12k USD.
no, these chips are optimized for inference not training & frankly cuda is still table stakes. HN loves it some Apple
Wouldn’t a cluster of M4 minis cost less and provide more VRAM? There are posts about people getting decent performance for a lot less than 12k USD.
Earlier quoted context omitted.
The GPU rental market is fairly reasonable. There's lots of companies doing it. (I work at one of them). 4x 4090 can be fetched for around $0.40/hour on some platforms ... about $1.20 on others depending on how available you want it. Regardless, all in, you can do an average 10-or-so-day train for If you want on-prem, wait a few months. The supply of 5000 series (probably announced at CES in a few days) should push m…
How soon could I break even on renting my GPUs out?
We aim at $1200/y for 3090, so around a year given descent electricity prices.
Highly recommend setting a lower power limit (usually 250W for 3090).
Earlier quoted context omitted.
Yes! His power supplies are 2x1500 Watt. That puts it at 3KW max which is more than a 20A circuit can provide (2400W). The standard outlet is typically rated at 15 amps or 1800W. And the 15A breaker is on one circuit. You can get 20A circuits but they need to be wired for it, and replacing the breaker won't cut it. Assuming his GPU is ~450W (his number) and power supplies are 80% efficient, well that means he's pulli…
In the US. The UK or EU will do you 3000W out of a standard domestic socket.
Earlier quoted context omitted.
Is there a reason you used hyperparameters rather than parameters? I was going to politely correct the terminology but you seem to be in AI for some time so either it was a mistype or I am misunderstanding what you are referencing.
It's a force of habit, parameters would be more accurate (almost everyone uses them interchangeably nowadays)
Earlier quoted context omitted.
It's a force of habit, parameters would be more accurate (almost everyone uses them interchangeably nowadays)
Wait what? Who actually calls trainable params "hyperparameters"? Nobody at OpenAI does, as far as I know.
Overall, it's something I've seen very often on social media and less technical articles about LLMs. OpenAI would fall into the "almost" category.