Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinybox – A powerful computer for deep learning

tinygrad.org

Post reply on HN
Print view View original

211–220 of 372 posts

Re: Tinybox – A powerful computer for deep learning

#211

Post by timschmidt »

segmondy wrote: post #200
timschmidt wrote: post #183

Earlier quoted context omitted.

Great option if you don't mind sharing your data with the cloud. Some businesses want to own the hardware their data resides on.

renting GPU, how is that sharing data with the cloud? you can rent GPU from GCP or AWS

I suppose if I rent a cloud GPU and just let it sit there dark and do nothing then I wouldn't have to move any data to it. Otherwise, I'm uploading some kind of work for it to do. And that usually involves some data to operate on. Even if it's just prompts.

Re: Tinybox – A powerful computer for deep learning

#212

Post by Aurornis »

bastawhiz wrote: post #32

There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…

> There's no way the red v2 is doing anything with a 120b parameter model.

I don't see the 120B claim on the page itself. Unless the page has been edited, I think it's something the submitter added.

I agree, though. The only way you're running 120B models on that device is either extreme quantization or by offloading layers to the CPU. Neither will be a good experience.

These aren't a good value buy unless you compare them to fully supported offerings from the big players.

It's going to be hard to target a market where most people know they can put together the exact same system for thousands of dollars less and have it assembled in an afternoon. RTX 6000 96GB cards are in stock at Newegg for $9000 right now which leaves almost $30,000 for the rest of the system. Even with today's RAM prices it's not hard to do better than that CPU and 256GB of RAM when you have a $30,000 budget.

Re: Tinybox – A powerful computer for deep learning

#213

Post by Aurornis »

mnkyprskbd wrote: post #209
alasdair_ wrote: post #208

I just don’t believe that this can run inference on a 120 billion parameter model at actually useful speeds. Obviously any Turing machine can run any size of model, so the “120B” claim doesn’t mean much - what actually matters is speed and I just don’t believe this can be speedy enough on models that my $5000 5090-based pc is too slow for and lacks enough vram for.

Look at the GPU and RAM spec; 120b seems workable.

For the red v2?

120B could run, but I wouldn't want to be the person who had to use it for anything.

To be fair, the 120B claim doesn't appear on the webpage. I don't know where it came from, other than the person who submitted this to HN

Re: Tinybox – A powerful computer for deep learning

#214

Post by mnkyprskbd »

Aurornis wrote: post #213
mnkyprskbd wrote: post #209

Earlier quoted context omitted.

Look at the GPU and RAM spec; 120b seems workable.

For the red v2? 120B could run, but I wouldn't want to be the person who had to use it for anything. To be fair, the 120B claim doesn't appear on the webpage. I don't know where it came from, other than the person who submitted this to HN

It is more than fair, also, you're comparing your 5k devices to 12k and more importantly 65k and >10m devices.

Re: Tinybox – A powerful computer for deep learning

#216

Post by sowbug »

p0w3n3d wrote: post #126

Quite expensive little bastard. I wonder how much does it make sense to invest in a such device, if you can get $0.40/mtok from hyperbolic for example

If you're OK letting them train on, and maybe keep, your data, then it's hard to beat cloud prices vs. local.

Re: Tinybox – A powerful computer for deep learning

#218

Post by Aurornis »

segmondy wrote: post #171
ericd wrote: post #146

Earlier quoted context omitted.

Was that cheaper than a Blackwell 6000? But yeah, 4x Blackwell 6000s are ~32-36k, not sure where the other $30k is going.

folks have too much money than sense, gpt-oss-120b full quant runs on my quad 3090 at 100tk/sec and that's with llama.cpp, with vllm it will probably run at 150tk/sec and that's without batching.

> gpt-oss-120b full quant runs on my quad 3090

A 120B model cannot fit on 4 x 24GB GPUs at full quantization.

Either you're confusing this with the 20B model, or you have 48GB modded 3090s.

Re: Tinybox – A powerful computer for deep learning

#219

Post by Aurornis »

mnkyprskbd wrote: post #214
Aurornis wrote: post #213

Earlier quoted context omitted.

For the red v2? 120B could run, but I wouldn't want to be the person who had to use it for anything. To be fair, the 120B claim doesn't appear on the webpage. I don't know where it came from, other than the person who submitted this to HN

It is more than fair, also, you're comparing your 5k devices to 12k and more importantly 65k and >10m devices.

The "to be fair" part of my comment was saying that the tinygrad website doesn't claim 120B.

Also nobody is comparing this box to an $10M nVidia rack scale deployment. They're comparing it to putting all of the same parts into their Newegg basket and putting it together themself.

Re: Tinybox – A powerful computer for deep learning

#220

Post by mellosouls »

Where is the 120B documented? This seems to be an editorialized title.

Edit: found a third party referencing the claim but it doesn't belong in the title here I think:

Meet the World’s Smallest ‘Supercomputer’ from Tiiny AI; A Machine Bold Enough to Run 120B AI Models Right in the Palm of Your Hand

https://wccftech.com/meet-the-worlds-smallest-supercomputer-...

Post reply on HN