141–150 of 372 posts
With 6 GPUs you have to deal with risers, pcie retimers, dual PSUs and custom case for so value proposition there was much better IMO
10 mil today... 1k in 10 years. Are OpenAI and Anthropic overvalued?
Earlier quoted context omitted.
You are still really limited in what you can run. So much stuff is cuda only.
Like what? Most of the good stuff is ported over already and anything else, tag Anush on X and see what you get. Also happy to help. The point is that they care now.
The problem with all these "AI box" startups is that the product is too expensive for hobbyists, and companies that need to run workloads at scale can always build their own servers and racks and save on the markup (which is substantial). Unless someone can figure out how to get cheaper GPUs & RAM there is really no margin left to squeeze out.
Earlier quoted context omitted.
If you’re spending $65,000 on this thing, needing two circuits seems like a minor problem
The $12,000 one also requires it.
There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…
But yeah, 4x Blackwell 6000s are ~32-36k, not sure where the other $30k is going.
The problem with all these "AI box" startups is that the product is too expensive for hobbyists, and companies that need to run workloads at scale can always build their own servers and racks and save on the markup (which is substantial). Unless someone can figure out how to get cheaper GPUs & RAM there is really no margin left to squeeze out.
The incremental price increases between products is funny. $12,000, $65,000, $10,000,000.
I was more worried by the 600kW power requirement... that's 200 houses at full load (3kw) in southern europe... which likely means 400 houses at half load. the town near my hometown has 650 – 800 houses (according to chatgpt). crazy.
A typical home just consumes rather little energy, now that LED lighting and heat pump cooling / heating became the norm.
The incremental price increases between products is funny. $12,000, $65,000, $10,000,000.
I was more worried by the 600kW power requirement... that's 200 houses at full load (3kw) in southern europe... which likely means 400 houses at half load. the town near my hometown has 650 – 800 houses (according to chatgpt). crazy.
Earlier quoted context omitted.
You are still really limited in what you can run. So much stuff is cuda only.
Like what? Most of the good stuff is ported over already and anything else, tag Anush on X and see what you get. Also happy to help. The point is that they care now.
But even in the amd stack things (like ck and aiter) consumer cards are not even second class citizens. They are a distance third at best. If you just want to run vllm with the latest model, if you can get it running at all there are going to be paper cuts all along the way and even then the performance won't be close to what you could be getting out of the hardware.