Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinybox – A powerful computer for deep learning

tinygrad.org

Post reply on HN
Print view View original

201–210 of 372 posts

Re: Tinybox – A powerful computer for deep learning

#201

Post by subscribed »

briandw wrote: post #197
0xbadcafebee wrote: post #159

Earlier quoted context omitted.

I think you're forgetting the wires? If you have one outlet with a 15-20A 120V circuit, then the wiring is almost certainly rated for 15-20A. If you just "combined" two 120V circuits into a 240V circuit, you still need an outlet that is rated for 30A, the wires leading to it also need to be rated for 30A, and it definitely needs a neutral. So you still need a new wire run if you don't have two 120V circuits right whe…

Correct me if I’m wrong, but doubling the volts doesn't change the amps, it doubles the watts. Watts = V*A.

Doubling the volts halves the amps. P = I * V indeed.

Re: Tinybox – A powerful computer for deep learning

#202

Post by ekropotin »

banana_giraffe wrote: post #195
ekropotin wrote: post #59

Earlier quoted context omitted.

I’m not very well versed in this domain, but I think it’s not going to be “VRAM” (GDDR) memory, but rather “unified memory”, which is essentially RAM (some flavour of DDR5 I assume). These two types of memory has vastly different bandwidth. I’m pretty curious to see any benchmarks on inference on VRAM vs UM.

A quick benchmark using float32 copies using torch cuda->cuda copies, comparing some random machines: Raptor Lake + 5080: 380.63 GB/s Raptor Lake (CPU for reference): 20.41 GB/s GB10 (DGX Spark): 116.14 GB/s GH200: 1697.39 GB/s This is a "eh, it works" benchmarks, but should give you a feel for the relative performance of the different systems. In practice, this means I can get something like 55 tokens a sec running…

Nice! Thanks for that.

55 t/s is much better than I could expect.

Re: Tinybox – A powerful computer for deep learning

#203

Post by jofzar »

ivraatiems wrote: post #12

There's some irony in the fact that this website reads as extremely NOT AI-generated, very human in the way it's designed and the tone of its writing. Still, this is a great idea, and one I hope takes off. I think there's a good argument that the future of AI is in locally-trained models for everyone, rather than relying on a big company's own model. One thought: The ability to conveniently get this onto a 240v circu…

Good? That's what I want out of all websites. I don't want to read what an AI believes is the best thing for a website, I want to know the honest truth.

Re: Tinybox – A powerful computer for deep learning

#204

Post by zach_vantio »

zozbot234 wrote: post #160
wmf wrote: post #156

Earlier quoted context omitted.

just want to run a 7-8b model locally This is already solved by running LM Studio on a normal computer.

Ollama or llama.cpp are also common alternatives. But a 8B model isn't going to have much real-world knowledge or be highly reliable for agentic workloads, so it makes sense that people will want more than that.

the compute density is insane. but giving a 70B model actual write access locally for agentic workloads is a massive liability. they still hallucinate too much. raw compute without strict state control is basically just a blast radius waiting to happen.

Re: Tinybox – A powerful computer for deep learning

#207

Post by amluto »

reactordev wrote: post #125
Salgat wrote: post #115

Earlier quoted context omitted.

It's more accurate to say that the typical 120V circuit is just a 240V source with the neutral tapped into the midpoint of the transformer winding.

This. It definitely comes in at a higher voltage.

Sort of? It’s 120V RMS to ground.

Re: Tinybox – A powerful computer for deep learning

#208

Post by alasdair_ »

I just don’t believe that this can run inference on a 120 billion parameter model at actually useful speeds.

Obviously any Turing machine can run any size of model, so the “120B” claim doesn’t mean much - what actually matters is speed and I just don’t believe this can be speedy enough on models that my $5000 5090-based pc is too slow for and lacks enough vram for.

Re: Tinybox – A powerful computer for deep learning

#209

Post by mnkyprskbd »

alasdair_ wrote: post #208

I just don’t believe that this can run inference on a 120 billion parameter model at actually useful speeds. Obviously any Turing machine can run any size of model, so the “120B” claim doesn’t mean much - what actually matters is speed and I just don’t believe this can be speedy enough on models that my $5000 5090-based pc is too slow for and lacks enough vram for.

Look at the GPU and RAM spec; 120b seems workable.

Re: Tinybox – A powerful computer for deep learning

#210

Post by amluto »

jcgrillo wrote: post #85
trollbridge wrote: post #27

Earlier quoted context omitted.

A typical U.S. 240V circuit is actually just two 120V circuits. Fairly trivial to rewire for that.

If you actually use two 120V circuits that way and one breaker flips the other half will send 120V through the load back into the other circuit. So while that circuit's breaker is flipped it is still live . Very bad. Much better to use a 240V breaker that picks up two rails in the panel.

I assume the device has two separate PSUs, each of which accepts 120-240V, and neither of which will backfeed its supply.
Post reply on HN