Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinybox – A powerful computer for deep learning

tinygrad.org

Post reply on HN
Print view View original

321–330 of 372 posts

Re: Tinybox – A powerful computer for deep learning

#321

Post by androiddrew »

oceanplexian wrote: post #40
bastawhiz wrote: post #32

There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…

It will work fine but it’s not necessarily insane performance. I can run a q4 of gpt-oss-120b on my Epyc Milan box that has similar specs and get something like 30-50 Tok/sec by splitting it across RAM and GPU. The thing that’s less useful is the 64G VRAM/128G System RAM config, even the large MoE models only need 20B for the router, the rest of the VRAM is essentially wasted (Mixing experts between VRAM and/System R…

Could you share what you are using for inference and how you are running it? I have a 64G VRAM/128G system RAM setup.

Re: Tinybox – A powerful computer for deep learning

#322

Post by datadrivenangel »

oceanplexian wrote: post #40
bastawhiz wrote: post #32

There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…

It will work fine but it’s not necessarily insane performance. I can run a q4 of gpt-oss-120b on my Epyc Milan box that has similar specs and get something like 30-50 Tok/sec by splitting it across RAM and GPU. The thing that’s less useful is the 64G VRAM/128G System RAM config, even the large MoE models only need 20B for the router, the rest of the VRAM is essentially wasted (Mixing experts between VRAM and/System R…

Yeah I've got the q4 gpt-oss-120b running at ~40-60 tokens per second on an M5 Pro.

Re: Tinybox – A powerful computer for deep learning

#323

Post by ericd »

Havoc wrote: post #306
ericd wrote: post #187

Earlier quoted context omitted.

How're you fitting a model made for 80 gig cards onto a GPU with 24 gigs at full quant?

He said quad 3090 not single

Yeah, pretty sure that was edited in after I commented because 150 toks/sec was also new, but could’ve just missed it.

Re: Tinybox – A powerful computer for deep learning

#324

Post by segmondy »

amarshall wrote: post #180
segmondy wrote: post #171

Earlier quoted context omitted.

folks have too much money than sense, gpt-oss-120b full quant runs on my quad 3090 at 100tk/sec and that's with llama.cpp, with vllm it will probably run at 150tk/sec and that's without batching.

You're almost certainly (definitely, in fact) confusing the 120b and 20b models.

I'm most certainly not doing so.

   seg@seg-epyc:~/models$ du -sh * /llmzoo/models/* | sort -n
   4.0K metrics.txt
   4.0K opus
   4.0K start_llama
   8.2G nvidia_Orchestrator-8B-Q8_0.gguf
   12K  config.ini
   34G  Qwen3.5-27B
   47G  Qwen3.5-35B
   51G  Qwen3.5-27B-BF16
   61G  gpt-oss-120b-F16.gguf
   65G  Qwen3.5-35B-BF16
   106G Qwen3.5-122B-Q6
   117G GLM4.6V
   175G MiniMax-M2.5
   232G /llmzoo/models/small_models
   240G Ernie4.5-300B
   377G DeepSeekv3.2-nolight
   380G /llmzoo/models/DeepSeek-V3.2-UD
   400G /llmzoo/models/Qwen3.5-397B-Q8
   424G /llmzoo/models/KimiK2Thinking
   443G DeepSeek-Math-v2
   443G DeepSeek-V3-0324-Q5
   500G /llmzoo/models/GLM5-Q5
   546G /llmzoo/models/KimiK2.5

Re: Tinybox – A powerful computer for deep learning

#325

Post by segmondy »

Aurornis wrote: post #218
segmondy wrote: post #171

Earlier quoted context omitted.

folks have too much money than sense, gpt-oss-120b full quant runs on my quad 3090 at 100tk/sec and that's with llama.cpp, with vllm it will probably run at 150tk/sec and that's without batching.

> gpt-oss-120b full quant runs on my quad 3090 A 120B model cannot fit on 4 x 24GB GPUs at full quantization. Either you're confusing this with the 20B model, or you have 48GB modded 3090s.

Some of you folks on here love to argue, gpt-oss-120b was trained in 4 bits, so it pretty much takes up 60gb.

Re: Tinybox – A powerful computer for deep learning

#326

Post by segmondy »

timschmidt wrote: post #211
segmondy wrote: post #200

Earlier quoted context omitted.

renting GPU, how is that sharing data with the cloud? you can rent GPU from GCP or AWS

I suppose if I rent a cloud GPU and just let it sit there dark and do nothing then I wouldn't have to move any data to it. Otherwise, I'm uploading some kind of work for it to do. And that usually involves some data to operate on. Even if it's just prompts.

So you also believe when you rent a server you are sharing your data with the cloud? AWS and GCP are copying all private data on servers? Give me a break. There's a big difference between renting a server and using an API.

Re: Tinybox – A powerful computer for deep learning

#328

Post by segmondy »

ericd wrote: post #193
zozbot234 wrote: post #190

Earlier quoted context omitted.

MoE layers offload to CPU inference is the easiest way, though a bit of a drag on performance

Yeah, I'd just be pretty surprised if they were getting 100 tokens/sec that way. EDIT: Either they edited that to say "quad 3090s", or I just missed it the first time.

you are correct, I did forget to add quad. you should join us in r/localllama

check out what other people are getting. you're welcome.

https://www.reddit.com/r/LocalLLaMA/comments/1nunq7s/gptoss1... https://www.reddit.com/r/LocalLLaMA/comments/1p4evyr/most_ec...

Re: Tinybox – A powerful computer for deep learning

#329

Post by Aurornis »

segmondy wrote: post #325
Aurornis wrote: post #218

Earlier quoted context omitted.

> gpt-oss-120b full quant runs on my quad 3090 A 120B model cannot fit on 4 x 24GB GPUs at full quantization. Either you're confusing this with the 20B model, or you have 48GB modded 3090s.

Some of you folks on here love to argue, gpt-oss-120b was trained in 4 bits, so it pretty much takes up 60gb.

Good point, but you still need KV cache and more. Fitting the model alone to RAM doesn’t get the job done.

Re: Tinybox – A powerful computer for deep learning

#330

Post by ericd »

segmondy wrote: post #328
ericd wrote: post #193

Earlier quoted context omitted.

Yeah, I'd just be pretty surprised if they were getting 100 tokens/sec that way. EDIT: Either they edited that to say "quad 3090s", or I just missed it the first time.

you are correct, I did forget to add quad. you should join us in r/localllama check out what other people are getting. you're welcome. https://www.reddit.com/r/LocalLLaMA/comments/1nunq7s/gptoss1... https://www.reddit.com/r/LocalLLaMA/comments/1p4evyr/most_ec...

Thanks for the confirmation, wasn't sure if I was just going a bit senile heh. Yeah, I love /r/localllama, some of the best actual practitioners of this stuff on the internet. Also, crazy awesome frankenrigs to try and get that many huge cards working together.

I was considering picking up a couple of the 48 gig 4090/3090s on an upcoming trip to China, but I just ended up getting one of the Max-Q's. But maybe the token throughput would still be higher with the 4090 route? Impressive numbers with those 3090s!

What's the rig look like that's hosting all that?

Post reply on HN