Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Serving AI from the Basement – 192GB of VRAM Setup

ahmadosman.com

Post reply on HN
Print view View original

271–279 of 279 posts

Re: Serving AI from the Basement – 192GB of VRAM Setup

#271

Post by matt-p »

raxxorraxor wrote: post #237
fragmede wrote: post #196

Earlier quoted context omitted.

your car is 12 volts, and USB is 5 volts; 12 or up to 20 these days for laptop charging. My computer's CPU is probably 1.8 volts but I can't remember the last time I had my multimeter on that, but that's still more than millivolts.

Probably meant milliampere, specifically 1 milliampere. But yes, usually lightweight engineers are familiar with TTL and limit themselves to 5V. 12V+ is another arcane realm you don't want to touch. Some old serial ports had 12V and a high max current. The DIY things you attached here were prone to kill your mainboard. Voltage/current is either 0 or 1. Anything higher kills software developers instantly.

In telecoms 48V dc is very common and not always even connectorised! It's "safe-ish" but DC makes me more nervous than 240v, big thick 400A cables into a rack are quite intimidating to see but the main issue is DC is sticky and doesn't have the safety protections of RCDs etc. Indeed you are lucky to get a working isolator.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#272

Post by wpietri »

matt-p wrote: post #270
wpietri wrote: post #208

Earlier quoted context omitted.

I'm not saying people shouldn't add breakers. I'm saying your talking like people are scaredy-cats and comparing it to working with toys or hello world is exactly the kind of of macho nonsense that leads people to do shoddy engineering. As a hobby, I restore pinball machines. A modern one is extremely careful about how it uses power, limiting wall current to a small, normally-sealed section of the machine. And even s…

To be fair I'd be quite a bit more relaxed working on 120v. Very supprised these machines don't run on dc internally?

In the older ones, it's almost all AC. One giant transformer, a couple of different voltages. Possibly with "high tap", a way to compensate for wall current with lower than expected voltages. The past is another country.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#273

Post by varispeed »

lbotos wrote: post #148
varispeed wrote: post #78

Earlier quoted context omitted.

You only "break" once...

I've been learning Japanese and a favorite of mine is: 一体 Which is used as "what the heck" but it's direct kanji translation is one body . https://jisho.org/word/%E4%B8%80%E4%BD%93

Fun fact as a kid I stuck my fingers in the loose mains wires as we were playing at unfinished building. The wires were live and I still remember it felt like it's going to break my arm. Fortunately I only got a slight burn. This got me interested in electronics which I started studying later in my life.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#274

Post by nathanasmith »

killingtime74 wrote: post #195

Did everyone just miss the fact that the post says the intention is to run Llama 3 405b but it has less than 1/4 of the VRAM required to do so? Did you just change your goals mid build? It's commonly known how much ram is required for a certain parameter size.

The system has 512 GB of RAM so while it'll be slower at inference, he really has about 704 GB at his disposal to run the model assuming he distributes the weights across the VRAM and system RAM.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#275

Post by nathanasmith »

orbital-decay wrote: post #108
mattnewton wrote: post #50

Earlier quoted context omitted.

The main thing stopping me from going beyond 2x 4090’s in my home lab is power. Anything around ~2k watts on a single circuit breaker is likely to flip it, and that’s before you get to the costs involved of drawing that much power for multiple days of a training run. How did you navigate that in a (presumably) residential setting?

>Anything around ~2k watts on a single circuit breaker is likely to flip it I'm curious, how do you use e.g. a washing machine or an electric kettle, if 2kW is enough to flip your breaker? You should simply know your wiring limits. Breaker/wiring at my home won't even notice this.

I heat water on the stove top which is plugged into a 240 volt outlet.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#276

Post by Tepix »

angoragoats wrote: post #247
metadat wrote: post #157

Earlier quoted context omitted.

Worth mentioning - this also cuts the available bandwidth to each card by 50%.

While you're technically correct, assuming you're using PCIe 4.0 or higher, the performance difference between x8 and x16 is practically zero.

Even for training?

Re: Serving AI from the Basement – 192GB of VRAM Setup

#277

Post by lostmsu »

1123581321 wrote: post #249
vunderba wrote: post #93

Earlier quoted context omitted.

I'm seeing this misunderstanding a lot recently. There's TWO components to putting together a viable machine learning rig: - Fitting models in memory - Inference / Training speed 8 x RTX 3090s will absolutely CRUSH a single Mac Studio in raw performance.

Crush by what factor?

80x-240x

Re: Serving AI from the Basement – 192GB of VRAM Setup

#278

Post by angoragoats »

Tepix wrote: post #276
angoragoats wrote: post #247

Earlier quoted context omitted.

While you're technically correct, assuming you're using PCIe 4.0 or higher, the performance difference between x8 and x16 is practically zero.

Even for training?

I haven’t done a ton of training but everything I’ve heard and read indicates that PCIe 4.0 8x provides enough bandwidth for just about any application. You might see a negligible drop in performance, but no more than a few percent.

Re: Serving AI from the Basement – 192GB of VRAM Setup

#279

Post by Tepix »

halJordan wrote: post #266
Tepix wrote: post #112

Earlier quoted context omitted.

If you merely want CUDA and lots of VRAM there‘s no reason to pick expensive 4090s over used 3090s

Well there is and it's called performance. You dont have to push your version of what an appropriate price/performance ratio is

Well, if you run out of VRAM, you drop off a performance cliff. That's a whole order of magnitude slower than just using a slower GPU but fitting everything into VRAM.
Post reply on HN