I always wonder about these expensive products: Does the company make them once its ordered or do they just make them beforehand?
He builds a batch every few months.
Tinybox – A powerful computer for deep learning
181–190 of 372 posts
Re: Tinybox – A powerful computer for deep learning
#182How does this thing cool down?
Re: Tinybox – A powerful computer for deep learning
#183Earlier quoted context omitted.
P40 is Tesla architecture which is no longer receiving driver or CUDA updates. And only available as used hardware. Fine for hobbyists, startups, and home labs, but there is likely a growing market of businesses too large to depend on used gear from ebay, but too small for a full rack solution from Nvidia. Seems like that's who they're targeting.
99% of interest is in inference. If you want to fine-tune a model, just rent the best gpu in the cloud. It's often cheaper and faster.
Great option if you don't mind sharing your data with the cloud. Some businesses want to own the hardware their data resides on.
Re: Tinybox – A powerful computer for deep learning
#184Earlier quoted context omitted.
If you actually use two 120V circuits that way and one breaker flips the other half will send 120V through the load back into the other circuit. So while that circuit's breaker is flipped it is still live . Very bad. Much better to use a 240V breaker that picks up two rails in the panel.
i am guessing, without any proof, that, when one breaker fails the server lose it all, or loose two GPUs, depending on whether one connected to the cpu side failed.
GPUs aren't electrically isolated from the motherboard though. An entire computer is a single unified power domain.
The only place where there's isolation is stuff like USB ports to avoid dangerous ground loop currents.
That said I believe the PSU itself provides full isolation and won't backfeed so using two on separate circuits should (maybe?) be safe. Although if one circuit tripped the other PSU would immediately be way over capacity. Hopefully that doesn't cause an extended brownout before the second one disables itself.
Re: Tinybox – A powerful computer for deep learning
#185Earlier quoted context omitted.
Was that cheaper than a Blackwell 6000? But yeah, 4x Blackwell 6000s are ~32-36k, not sure where the other $30k is going.
I bought the A100s used for a little over $6k each.
Oh, why'd you go that route? Considering going beyond 80 gigs with nvlink or something?
Re: Tinybox – A powerful computer for deep learning
#186Earlier quoted context omitted.
The $12,000 one also requires it.
Easier to get two circuits than rewire a breaker in an office you might be renting, no? (I work for an electrical contractor so my sense of ease might be overcorrecting)
And 240v is orders of magnitude more common worldwide than 120v
Re: Tinybox – A powerful computer for deep learning
#187Earlier quoted context omitted.
Was that cheaper than a Blackwell 6000? But yeah, 4x Blackwell 6000s are ~32-36k, not sure where the other $30k is going.
folks have too much money than sense, gpt-oss-120b full quant runs on my quad 3090 at 100tk/sec and that's with llama.cpp, with vllm it will probably run at 150tk/sec and that's without batching.
How're you fitting a model made for 80 gig cards onto a GPU with 24 gigs at full quant?
Re: Tinybox – A powerful computer for deep learning
#188Earlier quoted context omitted.
If you’re spending $65,000 on this thing, needing two circuits seems like a minor problem
they could had gone with the Max-Q version RTX PRO 6000 and only require 120V circuit. 10% performance hit, but half the power. fundamentally, looks like they are shipping consumer off-the-shelf hardwares in a custom box.
Yeah, the other big benefit is that the Max-Q's have blowers that exhaust the hot air out of the box, the workstation cards would each blow their exhaust straight into the intake of the card behind it. The last card in that chain would be cooking, as the air has already been heated up by 1800W, essentially a hair dryer on high.
Or could be the server edition 6000s that just have a heatsink and rely on the case to drive air through them, those are 600W cards.
Re: Tinybox – A powerful computer for deep learning
#189Earlier quoted context omitted.
99% of interest is in inference. If you want to fine-tune a model, just rent the best gpu in the cloud. It's often cheaper and faster.
Great option if you don't mind sharing your data with the cloud. Some businesses want to own the hardware their data resides on.
How many businesses have the capabilities and expertise to train their own models?
Re: Tinybox – A powerful computer for deep learning
#190Earlier quoted context omitted.
folks have too much money than sense, gpt-oss-120b full quant runs on my quad 3090 at 100tk/sec and that's with llama.cpp, with vllm it will probably run at 150tk/sec and that's without batching.
How're you fitting a model made for 80 gig cards onto a GPU with 24 gigs at full quant?
MoE layers offload to CPU inference is the easiest way, though a bit of a drag on performance