Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tinybox – A powerful computer for deep learning

tinygrad.org

Post reply on HN
Print view View original

271–280 of 372 posts

Re: Tinybox – A powerful computer for deep learning

#271

Post by gymbeaux »

$12,000 gets you 1Gb/s networking and vanilla Ubuntu 24.04. Napkin math on the hardware it looks like margins are around 50% which feels like a school fundraiser where everyone pays what is obviously way more than normal retail price for X because "it's for the children."

I'm not sure what tinygrad is but I assume the markup is because the customer is making a conscious choice to support the tinygrad project. But what's unusual is there is apparently no reason whatsoever to buy this hardware, even if you plan on using tinygrad exclusively for your project. At least with System76 hardware I get (in theory) first class support for Pop!_OS.

Re: Tinybox – A powerful computer for deep learning

#272

Post by aurareturn »

emidoots wrote: post #119
mayukh wrote: post #22

What’s the most effective ~$5k setup today? Interested in what people are actually running.

At $7.2k + tax: * RAM - $1500 - Crucial Pro 128GB Kit (2x64GB) DDR5 RAM, 5600MHz CP2K64G56C46U5, up to 4 sticks for 128GB or 256GB, Amazon * GPU - $4700 - RTX Pro 5000 48GB, Microcenter * CPU/Mobo bundle - $1100 - AMD Ryzen 7 9800X3D, MSI X870E-P Pro, ditch the 32GB RAM, Microcenter * Case - $220, Hyte Y70, Microcenter * Cooler - $155, Arctic Cooling Liquid Freezer III Pro, top-mount it, Microcenter * PSU - $180, RM1…

$7.2k just to run at best Qwen3.5-35B-A3B doesn't seem worth it at all.

This is certainly not the most effective use of $7k for running local LLMs.

The answer is a 16" M5 Max 128GB for $5k. You can run much bigger models than your setup while being an awesome portable machine for everything else.

Re: Tinybox – A powerful computer for deep learning

#273

Post by [deleted] »

Aurornis wrote: post #228
mellosouls wrote: post #220

Where is the 120B documented? This seems to be an editorialized title. Edit: found a third party referencing the claim but it doesn't belong in the title here I think: Meet the World’s Smallest ‘Supercomputer’ from Tiiny AI; A Machine Bold Enough to Run 120B AI Models Right in the Palm of Your Hand https://wccftech.com/meet-the-worlds-smallest-supercomputer-...

That third party link is from a different company (Tiiny with an extra i) Now I'm wondering if the HN title was submitted by some AI bot that couldn't tell the difference.

[deleted]

Re: Tinybox – A powerful computer for deep learning

#274

Post by mellosouls »

Aurornis wrote: post #228
mellosouls wrote: post #220

Where is the 120B documented? This seems to be an editorialized title. Edit: found a third party referencing the claim but it doesn't belong in the title here I think: Meet the World’s Smallest ‘Supercomputer’ from Tiiny AI; A Machine Bold Enough to Run 120B AI Models Right in the Palm of Your Hand https://wccftech.com/meet-the-worlds-smallest-supercomputer-...

That third party link is from a different company (Tiiny with an extra i) Now I'm wondering if the HN title was submitted by some AI bot that couldn't tell the difference.

Ha, good catch, I googled for Tinybox 120B and clearly didn't read the article beyond the seeming match.

Re: Tinybox – A powerful computer for deep learning

#276

Post by cgio »

jolmg wrote: post #199
Aperocky wrote: post #123

Earlier quoted context omitted.

I found it useful to preface with * this section written by me typing on keyboard * * this section produced by AI * And usually both exist in document and lengthy communications. This gets what I wanted across with exactly my intention and then I can attach 10x length worth of AI appendix that would be helpful indexing and references.

> attach 10x length worth of AI appendix that would be helpful indexing and references. Are references helpful when they're generated? The reader could've generated them themselves. References would be helpful if they were personal references of stuff you actually read and curated. The value then would be getting your taste. References from an AI may well be good-looking nonsense.

I agree wholeheartedly, I don’t see any balance in the effort someone dedicated to generating text vs me consuming it. If you feel there’s further insight to be gained by an llm, give me the prompt, not the output. Any communication channel reflects a balance of information content flowing and we are still adjusting to the proper etiquette.

Re: Tinybox – A powerful computer for deep learning

#277

Post by Tepix »

kristopolous wrote: post #56
mayukh wrote: post #22

What’s the most effective ~$5k setup today? Interested in what people are actually running.

Fully aware of the DGX spark I've actually been looking into AMD Ryzen AI Max+ 395/392 machines. There's some interesting things here like https://www.bee-link.com/products/beelink-gtr9-pro-amd-ryzen... and https://www.amazon.com/GMKtec-5-1GHz-LPDDR5X-8000MHz-Display... ... haven't pulled the trigger yet but apparently inferencing on these chips are not trash. Machines with the 4xx chips are coming next month so mayb…

4xx chips are less capable than the 395

Re: Tinybox – A powerful computer for deep learning

#278

Post by vkazanov »

roarcher wrote: post #259
fwipsy wrote: post #258

Earlier quoted context omitted.

Not a joke. It's just true.

It doesn't matter if it's a joke. The non-technical manager or VP making this purchase will not understand it and will expect poor treatment from this vendor, an expectation that will be reinforced by numerous other things on this page. There is no reason to include it at all.

It seems that you work a lot with managers who have no clue what they are buying and why.

I mean, you're not wrong: buying enterprise software from Oracle or Microsoft or Salesforce is pure pain.

But nobody expects buying niche hardware from a tiny vendor to involve the usual 128 pre/post sale meetings and 256 hours of professional services.

Also, relevant VP buying these things usually do understand the difference between AMD and Nvidia stacks really well. Like, really-really well.

Re: Tinybox – A powerful computer for deep learning

#280

Post by qubex »

paxys wrote: post #110

The problem with all these "AI box" startups is that the product is too expensive for hobbyists, and companies that need to run workloads at scale can always build their own servers and racks and save on the markup (which is substantial). Unless someone can figure out how to get cheaper GPUs & RAM there is really no margin left to squeeze out.

They’re kickstarting a TINY device that is pocketable and aimed at consumers. I’ve backed it (full disclosure).
Post reply on HN