Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinybox – A powerful computer for deep learning

tinygrad.org

Post reply on HN
Print view View original

191–200 of 372 posts

Re: Tinybox – A powerful computer for deep learning

#191

Post by adrianwaj »

ivraatiems wrote: post #12

There's some irony in the fact that this website reads as extremely NOT AI-generated, very human in the way it's designed and the tone of its writing. Still, this is a great idea, and one I hope takes off. I think there's a good argument that the future of AI is in locally-trained models for everyone, rather than relying on a big company's own model. One thought: The ability to conveniently get this onto a 240v circu…

"locally-trained models for everyone"

Wouldn't there be a massive duplication of effort in that case? It'll be interesting to see how the costs play out. There are security benefits to think about as well in keeping things local-first.

Re: Tinybox – A powerful computer for deep learning

#192

Post by timschmidt »

cootsnuck wrote: post #189
timschmidt wrote: post #183

Earlier quoted context omitted.

Great option if you don't mind sharing your data with the cloud. Some businesses want to own the hardware their data resides on.

How many businesses have the capabilities and expertise to train their own models?

No idea. Probably more every day.

Re: Tinybox – A powerful computer for deep learning

#193

Post by ericd »

zozbot234 wrote: post #190
ericd wrote: post #187

Earlier quoted context omitted.

How're you fitting a model made for 80 gig cards onto a GPU with 24 gigs at full quant?

MoE layers offload to CPU inference is the easiest way, though a bit of a drag on performance

Yeah, I'd just be pretty surprised if they were getting 100 tokens/sec that way.

EDIT: Either they edited that to say "quad 3090s", or I just missed it the first time.

Re: Tinybox – A powerful computer for deep learning

#194

Post by cmxch »

emidoots wrote: post #119
mayukh wrote: post #22

What’s the most effective ~$5k setup today? Interested in what people are actually running.

At $7.2k + tax: * RAM - $1500 - Crucial Pro 128GB Kit (2x64GB) DDR5 RAM, 5600MHz CP2K64G56C46U5, up to 4 sticks for 128GB or 256GB, Amazon * GPU - $4700 - RTX Pro 5000 48GB, Microcenter * CPU/Mobo bundle - $1100 - AMD Ryzen 7 9800X3D, MSI X870E-P Pro, ditch the 32GB RAM, Microcenter * Case - $220, Hyte Y70, Microcenter * Cooler - $155, Arctic Cooling Liquid Freezer III Pro, top-mount it, Microcenter * PSU - $180, RM1…

Surprised to see X3D given the reports of failures. I’ve opted for a regular 9900x and X670E-E just to have a bit more assurance.

Re: Tinybox – A powerful computer for deep learning

#195

Post by banana_giraffe »

ekropotin wrote: post #59
oofbey wrote: post #31

Earlier quoted context omitted.

DGX Spark is a fantastic option at this price point. You get 128GB VRAM which is extremely difficult to get at this price point. Also it’s a fairly fast GPU. And stupidly fast networking - 200gbps or 400gbps mellanox if you find coin for another one.

I’m not very well versed in this domain, but I think it’s not going to be “VRAM” (GDDR) memory, but rather “unified memory”, which is essentially RAM (some flavour of DDR5 I assume). These two types of memory has vastly different bandwidth. I’m pretty curious to see any benchmarks on inference on VRAM vs UM.

A quick benchmark using float32 copies using torch cuda->cuda copies, comparing some random machines:

    Raptor Lake + 5080: 380.63 GB/s
    Raptor Lake (CPU for reference): 20.41 GB/s
    GB10 (DGX Spark): 116.14 GB/s
    GH200: 1697.39 GB/s
This is a "eh, it works" benchmarks, but should give you a feel for the relative performance of the different systems.

In practice, this means I can get something like 55 tokens a sec running a larger model like gpt-oss-120b-Q8_0 on the DGX Spark.

Re: Tinybox – A powerful computer for deep learning

#197

Post by briandw »

0xbadcafebee wrote: post #159
trollbridge wrote: post #27

Earlier quoted context omitted.

A typical U.S. 240V circuit is actually just two 120V circuits. Fairly trivial to rewire for that.

I think you're forgetting the wires? If you have one outlet with a 15-20A 120V circuit, then the wiring is almost certainly rated for 15-20A. If you just "combined" two 120V circuits into a 240V circuit, you still need an outlet that is rated for 30A, the wires leading to it also need to be rated for 30A, and it definitely needs a neutral. So you still need a new wire run if you don't have two 120V circuits right whe…

Correct me if I’m wrong, but doubling the volts doesn't change the amps, it doubles the watts. Watts = V*A.

Re: Tinybox – A powerful computer for deep learning

#198

Post by roarcher »

paxys wrote: post #109
fhn wrote: post #107

"but if you haven't contributed to tinygrad your application won't be considered" this company expects people to work for free?

> See our bounty page to judge if you might be a good fit. Bounties pay you while judging that fit. Literally the line above that

They MIGHT pay you IF you're a fit. They're bounties, i.e. spec work. They also pay a max of $1000, most of them significantly less. You can see more info at the link in that line:

> All bounties paid out at my (geohot) discretion. Code must be clean and maintainable without serious hacks.

No thanks. If you want to try before you buy, have your candidates do a paid test project. Founders need to stop acting like it's a privilege to work for them. Any talent worth hiring has plenty of other options that will treat them with respect.

Re: Tinybox – A powerful computer for deep learning

#199

Post by jolmg »

Aperocky wrote: post #123
solarkraft wrote: post #114

Earlier quoted context omitted.

I find that the most respected writing about AI has very few signs of being written by AI. I'm guessing that's because people in the space are very sensitive to the signs and signal vs. noise.

I found it useful to preface with * this section written by me typing on keyboard * * this section produced by AI * And usually both exist in document and lengthy communications. This gets what I wanted across with exactly my intention and then I can attach 10x length worth of AI appendix that would be helpful indexing and references.

> attach 10x length worth of AI appendix that would be helpful indexing and references.

Are references helpful when they're generated? The reader could've generated them themselves. References would be helpful if they were personal references of stuff you actually read and curated. The value then would be getting your taste. References from an AI may well be good-looking nonsense.

Re: Tinybox – A powerful computer for deep learning

#200

Post by segmondy »

timschmidt wrote: post #183
segmondy wrote: post #179

Earlier quoted context omitted.

99% of interest is in inference. If you want to fine-tune a model, just rent the best gpu in the cloud. It's often cheaper and faster.

Great option if you don't mind sharing your data with the cloud. Some businesses want to own the hardware their data resides on.

renting GPU, how is that sharing data with the cloud? you can rent GPU from GCP or AWS
Post reply on HN