Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinybox – A powerful computer for deep learning

tinygrad.org

Post reply on HN
Print view View original

351–360 of 372 posts

Re: Tinybox – A powerful computer for deep learning

#351

Post by overfeed »

oofbey wrote: post #339
overfeed wrote: post #131

Earlier quoted context omitted.

> But moving from CUDA to ROCm is often more like a rewrite than a recompile. Isn't everyone* in this segment just using PyTorch for training, or wrappers like Ollama/vllm/llama.cpp for inference? None have a strict dependency on Cuda. PyTorch's AMD backend is solid (for supported platforms, and Strix Halo is supported). * enthusiasts whose budget is in the $5k range. If you're vendor-locked to CUDA, Mac Mini and Str…

Most everything starts as PyTorch. (Or maybe Jax.) But the inference engines all use hand tuned CUDA kernels - at least the good ones do. You have to do that to optimize things.

I'm certain inference engines don't use hand-tuned CUDA on Radeon or Mac Mini chips. My statement holds: those engines have no strict dependency on CUDA, or they'd be Nvidia-only.

Re: Tinybox – A powerful computer for deep learning

#352

Post by amluto »

razingeden wrote: post #236
amluto wrote: post #207

Earlier quoted context omitted.

Sort of? It’s 120V RMS to ground.

yes, this is accurate for US and “works” but it’s against code here. you’ll get mildly shocked by metallic cabinets and fixtures especially if you’re barefoot and become the new shortest path to ground. old construction in the US sometimes did this intentionally (so old, the house didn’t have grounds. Or to “pass” an inspection and sell a place) but if a licensed electrician sees this they have to fix it. I’m dealing…

Late reply: I think you misunderstood my comment. I was replying to:

> It definitely comes in at a higher voltage.

The voltage supplied to a US house is 120V RMS measured phase-to-ground. You will not find a higher voltage in your house. This does not mean that it’s appropriate to run any non-negligible current from phase to the ground (green / “equipment grounding conductor”) wires.

One can get vaguely close to an accurate understanding by imagining that there are four wires coming out of your main panel: +120V, -120V, 0V white (the “actually use me” wire) and 0V green (a safety wire where any current more than a few mA or maybe tens of mA depending on application is at least a mistake). There’s no 240V to be found.

This explanation falls apart pretty quickly — the US system is AC, not DC.

Re: Tinybox – A powerful computer for deep learning

#353

Post by vlovich123 »

lostmsu wrote: post #45
embedding-shape wrote: post #16

Earlier quoted context omitted.

Yeah, and labeling AMD "Driver Quality" as "Good" (for comparison, they label nvidia's driver quality as "Great").

Things changed. On my new Ryzen Strix Halo laptop I was able to run training experiments with PyTorch on Windows day 1: https://news.ycombinator.com/item?id=46052535

Yeah, installing random wheels from non official sources is an improvement. Not sure I’d characterize that as an unmitigated win. But also as soon as you try to do more involved things, at least personally, I ran into serious challenges getting things to work.

Re: Tinybox – A powerful computer for deep learning

#354

Post by lofaszvanitt »

Havoc wrote: post #307
roarcher wrote: post #221

> In order to keep prices low and quality high, we don't offer any customization to the box or ordering process. If you aren't capable of ordering through the website, I'm sorry but we won't be able to help. Has this guy never worked on a B2B product before? Nobody is going to order a $10 million piece of infrastructure through your website's order form. And they are definitely going to want to negotiate something ,…

> arrogant to the point of being almost hostile. The YouTube rap video of geohotz telling Sony lawyers suing him to blow him is still up. His style of dealing with corporate matters is certainly unconventional

Well, at least he had the power that average joes don't have. And he used it well.

Re: Tinybox – A powerful computer for deep learning

#355

Post by ColonelPhantom »

Aurornis wrote: post #329
segmondy wrote: post #325

Earlier quoted context omitted.

Some of you folks on here love to argue, gpt-oss-120b was trained in 4 bits, so it pretty much takes up 60gb.

Good point, but you still need KV cache and more. Fitting the model alone to RAM doesn’t get the job done.

GPT-OSS is tailored to be extremely memory efficient. Not only is it natively using the 4.25 bit per token MXFP4 format, but it also uses sliding window attention for half of its layers. It also doesn't have that many layers, only 36 for the 120B version and 24 for the 120B version. (The 120B is also much much sparser than the 20B.)

I found a Reddit comment claiming only 36 KiB per token. With that, half a million tokens fits in 18 GB, which is less than one GPU. And three GPUs fit the parameters with room to spare (64 out of 72 GB).

Re: Tinybox – A powerful computer for deep learning

#356

Post by bastawhiz »

ericd wrote: post #185
bastawhiz wrote: post #176

Earlier quoted context omitted.

I bought the A100s used for a little over $6k each.

Oh, why'd you go that route? Considering going beyond 80 gigs with nvlink or something?

When the costs come down, I'll add two H100s. Until I have more work to saturate the GPUs, they're really at the limit of what I can make time to use them for. Give me a year of writing code and I'll have the need!

Re: Tinybox – A powerful computer for deep learning

#357

Post by bastawhiz »

packetlost wrote: post #345
bastawhiz wrote: post #32

There's no way the red v2 is doing anything with a 120b parameter model. I just finished building a dual a100 ai homelab (80gb vram combined with nvlink). Similar stats otherwise. 120b only fits with very heavy quantization, enough to make the model schizophrenic in my experience. And there's no room for kv, so you'll OOM around 4k of context. I'm running a 70b model now that's okay, but it's still fairly tight. And…

This does not match my experience with 120B~ models. I run Qwen3.5 122b A10B on about 80GB of vRAM just fine.

Qwen 3.5 is MoE. But you're also almost certainly running a quantized version. 120B is well over 200gb at bf16. With int4 you're looking at 60gb or so. Qwen uses relatively little kv (only about 2gb for 64k context). So you're not too snug, but if qwen isn't cutting it for you, as it didn't for me, you're kind of in a pickle. For writing tasks, int4 was simply too chaotic. I also couldn't get it to use tools.

For me, qwen didn't cut it. You're not fine tuning a 120b parameter model with 80gb. You're probably not going to be able to abliterate it either, because it's moe. Other options use more vram, and where you'd have a fair amount of buffer with qwen, you're pressed with other big models.

Re: Tinybox – A powerful computer for deep learning

#359

Post by fwipsy »

roarcher wrote: post #259
fwipsy wrote: post #258

Earlier quoted context omitted.

Not a joke. It's just true.

It doesn't matter if it's a joke. The non-technical manager or VP making this purchase will not understand it and will expect poor treatment from this vendor, an expectation that will be reinforced by numerous other things on this page. There is no reason to include it at all.

To me it signals honesty. But this is a subjective judgement. It really sounds like you subjectively disliked the page, and you're trying to present that dislike as objective fact. It really annoyed me the way you kept changing your argument to justify that. Why not just say "I dislike their marketing copy, it rubbed me the wrong way" and leave it at that?

Re: Tinybox – A powerful computer for deep learning

#360

Post by crossroadsguy »

phrotoma wrote: post #304
roarcher wrote: post #221

> In order to keep prices low and quality high, we don't offer any customization to the box or ordering process. If you aren't capable of ordering through the website, I'm sorry but we won't be able to help. Has this guy never worked on a B2B product before? Nobody is going to order a $10 million piece of infrastructure through your website's order form. And they are definitely going to want to negotiate something ,…

> arrogant to the point of being almost hostile First encounter with geohot eh?

What does this mean? Is it some reference to different temperaments across geographies? Or some Internet slang?
Post reply on HN