Earlier quoted context omitted.
[flagged]
A single 3090 will deliver more tflops than the m2 ultra.
Yep. The 8x3090 setup should be at least 10 times faster than the Mac.
Serving AI from the Basement – 192GB of VRAM Setup
11–20 of 279 posts
Re: Serving AI from the Basement – 192GB of VRAM Setup
#12What is the power draw under load/idle? Does it noticeably increase the room temperature? Given the surroundings (aka the huge pile of boxes behind the setup), curious if you could get away with just a couple of box fans instead of the array of case fans.
Are you intending to use the capacity all for yourself or rent it out to others?
Re: Serving AI from the Basement – 192GB of VRAM Setup
#13Did you write this with the LLM running on the rig?
Re: Serving AI from the Basement – 192GB of VRAM Setup
#14Re: Serving AI from the Basement – 192GB of VRAM Setup
#15this is why we need an actual AI blockchain, so we can donate GPU and earn rewards for the p2p api calls using the distributed model.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#16I thought I was balling with my dual 3090 with nvlink. I haven’t quite yet figured out what to do with 48GB VRAM yet.
I hope this guy posts updates.
Re: Serving AI from the Basement – 192GB of VRAM Setup
#17this is why we need an actual AI blockchain, so we can donate GPU and earn rewards for the p2p api calls using the distributed model.
> donate GPU .. earn rewards
Is a blockchain needed to sell unused GPU capacity?
Re: Serving AI from the Basement – 192GB of VRAM Setup
#18Re: Serving AI from the Basement – 192GB of VRAM Setup
#19Earlier quoted context omitted.
[flagged]
A single 3090 will deliver more tflops than the m2 ultra.
The 3090s also give you access to all those CUDA inference and training frameworks used in industry. With Apple you're pretty much stuck with either MLX or llama.cpp.
Honestly though I'd be curious to see a cost analysis of Apple vs. Nvidia for commercial batched inference. The Nvidia system can obviously spit out more tokens/s but for the same price you could have multiple Mac Studios running the same model (and users would be dispatched to one of them).
Re: Serving AI from the Basement – 192GB of VRAM Setup
#20You could just buy a Mac Studio for 6500 USD, have 192 GB of unified RAM and have way less power consumption.
Are people running llama 3.1 405B on them?
I'm running 70B models (usually in q4 .. q5_k_m, but possible to q6) on my 96Gbyte Macbook Pro with M2-Max (12 cpu cores, 38 gpu cores). This also leaves me with plenty of ram for other purposes.
I'm currently using reflection:70b_q4 which does a very good job in my opinion. It generates with 5.5 tokens/s for the response, which is just about my reading speed.
edit: I usually dont run larger models (q6) because of the speed. I'd guess a 405B model would just be awfully slow.