Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All You Need Is 4x 4090 GPUs to Train Your Own Model

sabareesh.com

Post reply on HN
Print view View original

111–120 of 125 posts

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#111

Post by whimsicalism »

Bancakes wrote: post #71
minimaxir wrote: post #10

Earlier quoted context omitted.

The bottleneck for most model training sizes is VRAM, and since each 4090 has 24 GB VRAM, that's 96 GB VRAM total. The article mentions that it can train LLMs from scratch up to 1 billion hyperparameters, which tracks. Nowadays that's not a lot: a single H100 that you can now rent has 80 GB VRAM, and doesn't have the technical overhead of handling work across GPUs.

I doubt the RAM is added up. I think that’s only a feature reserved for their NVLinked HPC series cards. In fact, without nvlink, I don’t see how you’d connect them together to compute a single task in a performant and efficient way.

yeah essentially this

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#112

Post by [deleted] »

minimaxir wrote: post #92
llm_nerd wrote: post #84

Earlier quoted context omitted.

Is there a reason you used hyperparameters rather than parameters? I was going to politely correct the terminology but you seem to be in AI for some time so either it was a mistype or I am misunderstanding what you are referencing.

It's a force of habit, parameters would be more accurate (almost everyone uses them interchangeably nowadays)

[deleted]

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#113

Post by llm_nerd »

minimaxir wrote: post #100
unixpickle wrote: post #99

Earlier quoted context omitted.

Wait what? Who actually calls trainable params "hyperparameters"? Nobody at OpenAI does, as far as I know.

People who are making quick social media posts while taking a casual walk outside on websites that don't make it easy to edit posts and are not expecting to be nitpicked about it. Overall, it's something I've seen very often on social media and less technical articles about LLMs. OpenAI would fall into the "almost" category.

It's okay to say that you mistyped or whatever, while taking a casual walk outside on websites that don't make it easy to edit posts and are not expected to be nitpicked about it. Throwing in that everyone uses them interchangeably, however, is just profoundly wrong on every level.

I wasn't nitpicking. It is a HUGE differentiation, and I pointed it out specifically because people pick up on terminology so people who might not know better will go forward and just drop in the more super duper hyperparameter, not realizing that it makes them look like they don't know what they're talking about. As I said in the other post, no one who knows anything uses them interchangeably. It is just completely wrong.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#114

Post by amluto »

belter wrote: post #42
gzer0 wrote: post #30

This is a great build, thanks for sharing your learnings. The best build I have seen so far had 6x4090's. Video: https://www.youtube.com/watch?v=C548PLVwjHA Specifications - GPU Accelerator - 6 x 24GB NVIDIA GeForce RTX 4090 - Processor - Intel Xeon W7-3465X, 28C/56T, 2.5GHz - 4.8GHz - Memory - 256GB (8x32GB) DDR5 ECC 4800MHz - System Drive - 2TB Samsung 980 PRO NVMe PCIe 4.0 M.2 SSD - Storage Drive - 4TB Samsung 870…

Don't forget to talk to your local power company one year in advance. They will need to upgrade your local substation transformer... :-)

This build is 3kVA max. That’s about 1/3 of a current gen EV, only 15% of an original Tesla Model S with dual chargers, and about equal to a standard American oven. This is much more polite to the grid than, say, a couple of tea kettles or especially a reasonably sized electric tankless water heater.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#115

Post by MuffinFlavored »

magicalhippo wrote: post #91

On a tangent, if I wished to fine-tune one of those medium sized models like Gemma2 9B or Llama 3.2 Vision 11B, what kind of hardware would I need and how would I go about it? I see a lot of guides but most focus on getting the toolchain up and running, and not much talk about what kind of dataset do I need to do a good fine tuning. Any pointers appreciated.

What would you expect from fine tuning? What would the input training material be, and what would the expected differences in output be?

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#116

Post by minimaxir »

llm_nerd wrote: post #113
minimaxir wrote: post #100

Earlier quoted context omitted.

People who are making quick social media posts while taking a casual walk outside on websites that don't make it easy to edit posts and are not expecting to be nitpicked about it. Overall, it's something I've seen very often on social media and less technical articles about LLMs. OpenAI would fall into the "almost" category.

It's okay to say that you mistyped or whatever, while taking a casual walk outside on websites that don't make it easy to edit posts and are not expected to be nitpicked about it. Throwing in that everyone uses them interchangeably, however, is just profoundly wrong on every level. I wasn't nitpicking. It is a HUGE differentiation, and I pointed it out specifically because people pick up on terminology so people who…

Again, I've heard and used the terminology "model hyperparameter" in place of "model parameter", and I've also heard "model parameter" in place of "model hyperparameter" because not every human interaction is a paper on arXiv and the terms are obviously very similar. The context of the term is what matters in the end (as demonstrated by other comments following my correct intent), and society will not crumble if using either term incorrectly in casual conversation. No one intentionally uses the wrong term, but as jokingly said in another comment "when you get really deep into model training, it can seem like there are a billion hyperparameters you have to worry about."

I appreciate being corrected, but you are the one who asked for my opinion based on my extensive time in AI, you can choose to believe it or not.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#117

Post by kristopolous »

lostmsu wrote: post #97
yieldcrv wrote: post #57

Earlier quoted context omitted.

How soon could I break even on renting my GPUs out?

If you are on Windows, take a look at https://borg.games/setup (founder here) We aim at $1200/y for 3090, so around a year given descent electricity prices. Highly recommend setting a lower power limit (usually 250W for 3090).

Btw, for other people reading this, the main player in the "rentable gamer gpu" space is salad.com who 6 months ago cut a deal with civitai (https://blog.salad.com/civitai-salad/). They're trying to capture enterprise customers to use the extra cycles on teenager's gaming rigs.

The industry is full of effectively "imitation companies" right now. For instance, runpod, quickpod, simplepod and clore are the ones cloning us at vast right now.

We see them in our discord, they try to snipe away customers, get in our comment threads on reddit and twitter with self-promotes, clone our features ... this is the ferocious wild west days of this industry. I've even gotten personal emails from a few who I guess scanned their database looking for registration addresses from other companies in the space.

There's even companies like primeintellect which are trying to become the market of markets - but they have their own program - it's clearly a play to snipe other customers by funneling them through some interface where they'll eventually push out the other companies and promote their own instances.

Then there's interesting insider hype players with their own infra like sfcompute who are trying to pretend like they invented interruptible instances and somehow get a bunch of people treating them like they're innovators. The resellable contracts they talk about are a pretty common feature and especially from the host's programmatic command line controller, it's just usually tucked deep in the documentation. They're doing effectively a re-prioritization play.

I guess my angle is "highest integrity possible". It's certainly a gamble - scammy companies sometimes capture a market then become unscammy - I'll hold my tongue but there's plenty of examples.

It's interesting times.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#118

Post by magicalhippo »

MuffinFlavored wrote: post #115
magicalhippo wrote: post #91

On a tangent, if I wished to fine-tune one of those medium sized models like Gemma2 9B or Llama 3.2 Vision 11B, what kind of hardware would I need and how would I go about it? I see a lot of guides but most focus on getting the toolchain up and running, and not much talk about what kind of dataset do I need to do a good fine tuning. Any pointers appreciated.

What would you expect from fine tuning? What would the input training material be, and what would the expected differences in output be?

In several cases I've been wanting better prompt adherence.

Llama 3.2 Vision is very strictly trained to output a summary at the end which I find difficult to get it stop doing for example.

Another one is that when given a math problem and asked to generate some code that computes the result, most models outputs code fine but insists on doing calculations themselves even if the prompt explicitly say they shouldn't. As expected, sometimes these intermediate calculations are incorrect and hence I don't want the LLM to do that when the produced code would handle it perfectly. If the input prompt contains "four times five" I want the model to generate "4 * 5" rather than "20", consistently.

I've been curious to see if I could tune them to adhere better to the kind of prompts I would be giving.

For LLama 3.2 Vision I've also been curios if I can get it to focus on different details when asked to describe certain images. In many cases it is great but sometimes misses some key aspects.

As for the input training material, that's what I'm trying to figure out what I need. I feel a lot of the guides are like that "how to draw an owl" meme[1], leaving out some crucial aspects of the whole process. Obviously I need input prompts and expected answers, but how many, how much variation on each example, and do I need to include data it was already trained on to avoid overfitting or something like that? None of the guides I've found so far touch on these aspects.

[1]: https://knowyourmeme.com/memes/how-to-draw-an-owl

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#119

Post by halyconWays »

sabareesh wrote: post #60
halyconWays wrote: post #39

Why not 3090s? Same VRAM and cheaper. With both setups you'd be limited to 1B. By contrast, you can run 4-bit quants of Llama 70B on two {3,4}090s, and it's still pretty lobotomized by modern standards. You can also train your own model even without GPUs. Just depends on parameter size.

It is previous architecture and it doesnt support newer version of Flash Attention , fp8 training etc

That's fair. I did run into that issue when trying to speed up Hunyuan

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#120

Post by dkkergoog »

amluto wrote: post #114
belter wrote: post #42

Earlier quoted context omitted.

Don't forget to talk to your local power company one year in advance. They will need to upgrade your local substation transformer... :-)

This build is 3kVA max. That’s about 1/3 of a current gen EV, only 15% of an original Tesla Model S with dual chargers, and about equal to a standard American oven. This is much more polite to the grid than, say, a couple of tea kettles or especially a reasonably sized electric tankless water heater.

[dead]
Post reply on HN