Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinybox – A powerful computer for deep learning

tinygrad.org

Post reply on HN
Print view View original

161–170 of 372 posts

Re: Tinybox – A powerful computer for deep learning

#161

Post by ycui1986 »

wat10000 wrote: post #23
ivraatiems wrote: post #12

There's some irony in the fact that this website reads as extremely NOT AI-generated, very human in the way it's designed and the tone of its writing. Still, this is a great idea, and one I hope takes off. I think there's a good argument that the future of AI is in locally-trained models for everyone, rather than relying on a big company's own model. One thought: The ability to conveniently get this onto a 240v circu…

If you’re spending $65,000 on this thing, needing two circuits seems like a minor problem

they could had gone with the Max-Q version RTX PRO 6000 and only require 120V circuit. 10% performance hit, but half the power.

fundamentally, looks like they are shipping consumer off-the-shelf hardwares in a custom box.

Re: Tinybox – A powerful computer for deep learning

#162

Post by wmf »

throwatdem12311 wrote: post #86
flumpcakes wrote: post #77

Earlier quoted context omitted.

These specs look enormously cheaper than doing it with dell servers. The last quote I had for a bog standard dell server was $50k and only if bought in the next few days or so. The prices are going up weekly.

So what’s the catch? If it seems too good to be true it probably is.

These are "unsupported" configurations. Nvidia/AMD discourage running multiple gaming/workstation cards and encourage customers to buy $500K SXM/OAM servers.

Re: Tinybox – A powerful computer for deep learning

#163

Post by ycui1986 »

isatty wrote: post #79
ivraatiems wrote: post #72

Earlier quoted context omitted.

The $12,000 one also requires it.

Surprisingly affordable but I’m not really interested in the 9070XT. If it shipped with like 4090+ (for a higher price) it’d be more tempting.

9070XT provide roughly same inference performance at double the power, half the cost, as RTX PRO 4500. So this one is optimized for total BOM cost.

Re: Tinybox – A powerful computer for deep learning

#164

Post by llbbdd »

wongarsu wrote: post #2

Sound like solid prebuilt with well balanced components and a pretty case Not revolutionary in any way, but nice. Unless I'm missing something here?

If you wanted a box built by geohot, most recently known for signing on to Elons Twitter and then bailing, it's for you

Re: Tinybox – A powerful computer for deep learning

#165

Post by ycui1986 »

jcgrillo wrote: post #85
trollbridge wrote: post #27

Earlier quoted context omitted.

A typical U.S. 240V circuit is actually just two 120V circuits. Fairly trivial to rewire for that.

If you actually use two 120V circuits that way and one breaker flips the other half will send 120V through the load back into the other circuit. So while that circuit's breaker is flipped it is still live . Very bad. Much better to use a 240V breaker that picks up two rails in the panel.

i am guessing, without any proof, that, when one breaker fails the server lose it all, or loose two GPUs, depending on whether one connected to the cpu side failed.

Re: Tinybox – A powerful computer for deep learning

#166

Post by ottah »

bastawhiz wrote: post #32

There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…

Honestly two rtx 8000s would probably have a better return on investment than the red v2. I have an eight gpu server, five rtx 8000, three rtx 6000 ada. For basic inference, the 8000s aren't bad at all. I'm sure the green with four rtx pro 6000s are dramatically faster, but there's a $25k markup I don't honestly understand.

Re: Tinybox – A powerful computer for deep learning

#167

Post by ottah »

overfeed wrote: post #124
bastawhiz wrote: post #32

There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…

> I'm also confused why this is 12U. My whole rig is 4u. I imagine that's because they are buying a single SKU for the shell/case. I imagine their answer to your question would be: In order to keep prices low and quality high, we don't offer any customization to the server dimensions

That's just such a massively oversized server for the number of gpus. It's not like they're doing anything special either. I can buy an appropriately sized supermicro chassis myself and throw some cards in it. They're really not adding enough value add to overspend on anything.

Re: Tinybox – A powerful computer for deep learning

#169

Post by segmondy »

alexfromapex wrote: post #168

$12,000 for the base model is insane. I have an Apple M3 Max with 128GB RAM that can run 120B parameter models using like 80 watts of electricity at about 15-20 tokens/sec. It's not amazing for 120B parameter models but it's also not 12 grand.

it's for fools. i bought 160gb of vram for $1000 last year. 96gb of p40 VRAM can be had for under $1000. And it will run gpt-oss-120b Q8 at probably 30tk/sec
Post reply on HN