Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All You Need Is 4x 4090 GPUs to Train Your Own Model

sabareesh.com

Post reply on HN
Print view View original

101–110 of 125 posts

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#101

Post by pilooch »

magicalhippo wrote: post #91

On a tangent, if I wished to fine-tune one of those medium sized models like Gemma2 9B or Llama 3.2 Vision 11B, what kind of hardware would I need and how would I go about it? I see a lot of guides but most focus on getting the toolchain up and running, and not much talk about what kind of dataset do I need to do a good fine tuning. Any pointers appreciated.

I do this for many application. 2 to 4 RTXA5000 do the job (Lora finetune). As for dataset, depending on your task, you need image / text pairs.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#102

Post by magicalhippo »

pilooch wrote: post #101
magicalhippo wrote: post #91

On a tangent, if I wished to fine-tune one of those medium sized models like Gemma2 9B or Llama 3.2 Vision 11B, what kind of hardware would I need and how would I go about it? I see a lot of guides but most focus on getting the toolchain up and running, and not much talk about what kind of dataset do I need to do a good fine tuning. Any pointers appreciated.

I do this for many application. 2 to 4 RTXA5000 do the job (Lora finetune). As for dataset, depending on your task, you need image / text pairs.

> As for dataset, depending on your task, you need image / text pairs.

I guess the main question is, do you just prepare training data as if you were training from scratch, or is there some particularities to finetuning that should be considered?

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#103

Post by bb88 »

bpye wrote: post #94
bb88 wrote: post #53

Earlier quoted context omitted.

In the UK they get that by doubling the voltage. The current draw will still be similar to the US. It's over current that causes fires.

The _power_ draw will be similar, but a 13A 230V outlet can do 2990W, vs a 15A 110V outlet at 1650W.

You proved my argument! Lol.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#104

Post by KeplerBoy »

Dylan16807 wrote: post #90
KeplerBoy wrote: post #40

Earlier quoted context omitted.

The thing is you need a threadripper-class build to make use of 4 GPUs in the first place. Ordinary PCs don't have the PCIe lanes necessary for that. But pricing is okay-ish, have a look at Geohot's Tinybox for turnkey solutions.

How much PCIe bandwidth do you need to avoid it being the bottleneck?

Depends on the Application. In Bitcoin farming it famously was not an issue at all, manufacturers came up with the weirdest motherboards featuring many x1 pcie slots. Look up the Biostar TB360-BTC PRO 2.0 if you want to see a curiosity.

In Deep Learning it depends on your sharding strategy.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#105

Post by rldjbpin »

nice writeup, but i feel that for most people, the software side of training models should be more interesting and accessible.

for one, "full" gpu utilization, one or many, remains an open topic in training workflows. spending efforts towards that, while renting from cloud, is a more accessible and fruitful to me than to finetune for marginal improvements.

this course was a nice source of inspiration - https://efficientml.ai/ - and i highly recommend looking into this to see what to do next with whatever hardware you have to work with.

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#108

Post by dogma1138 »

FridgeSeal wrote: post #76
manmal wrote: post #41

Earlier quoted context omitted.

On Reddit there's reports of 8x4090, or even 8xH100. I don't know where people get this kind of money for this, and why they don't rent infra instead.

Probably because they are after a lot of fast, local storage, and _that_ is where rented ML infra providers will sting you. Edit: could also just be more-money-than-sense. Never discount stupidity.

Hardware can be resold, and bought 2nd hand also.

The 4090 will likely maintain 50% of its current value due to its memory capacity over the next 12-18 months.

CapEx vs OpEx is a thing even if you are not a business…

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#109

Post by KuriousCat »

manmal wrote: post #41
gzer0 wrote: post #30

This is a great build, thanks for sharing your learnings. The best build I have seen so far had 6x4090's. Video: https://www.youtube.com/watch?v=C548PLVwjHA Specifications - GPU Accelerator - 6 x 24GB NVIDIA GeForce RTX 4090 - Processor - Intel Xeon W7-3465X, 28C/56T, 2.5GHz - 4.8GHz - Memory - 256GB (8x32GB) DDR5 ECC 4800MHz - System Drive - 2TB Samsung 980 PRO NVMe PCIe 4.0 M.2 SSD - Storage Drive - 4TB Samsung 870…

On Reddit there's reports of 8x4090, or even 8xH100. I don't know where people get this kind of money for this, and why they don't rent infra instead.

Why do you think RoI is better when infra is rented?

Re: All You Need Is 4x 4090 GPUs to Train Your Own Model

#110

Post by whimsicalism »

kristopolous wrote: post #54
whimsicalism wrote: post #44

Earlier quoted context omitted.

don't you need nvlink? feel like an 80gb a100 would start being worth it at a $1.20/4x 4090 price point

Some suppliers have support for it, some don't. They either use docker or kvm and it depends on how clever their hosting software is. We can do it, but that's a recent thing. it's really hit or miss

? sorry i really don't understand this reply... some suppliers have support for nvlink on 4090? i doubt that
Post reply on HN