Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"Tinyboxes finally have a buy it now button"

twitter.com

Post reply on HN
Print view View original

81–90 of 138 posts

Re: "Tinyboxes finally have a buy it now button"

#81

Post by andersa »

michaelt wrote: post #80
andersa wrote: post #64

Earlier quoted context omitted.

These are not redundant PSUs, each PSU powers different GPUs in the same machine. Are you sure connecting them to different phases is a good idea? I've been looking for a proper answer to this for a while, because I want to build a similar machine with 8 GPUs (~4500W max load) which would need to be split between two 16A 230V circuits.

The transformers in the power supplies provide 'isolation' between the input and output - which means you can connect the outputs together, even when the inputs are on different phases. Are you planning to build such a machine for your personal home use? If so you should be aware that (a) you might find server hardware hasn't thoroughly tested compatibility with things like suspend; (b) you might find games haven't t…

It's for personal use, though it would not run any games, it would be for running offline inference and other experiments. Probably not a smart purchase, but a fun one...

That is good to know multiple phases can work. Perhaps there would still be a fire risk in case of a short? Like somehow bridging the circuits > breakers don't trip?

Re: "Tinyboxes finally have a buy it now button"

#82

Post by anon389r58r58 »

StrangeDoctor wrote: post #78
andersa wrote: post #56

Earlier quoted context omitted.

You can't fine tune a 70B model with this. It barely even fits the weights before it runs out of vram. Need a bigger machine.

I think the idea is you network them together if you need more and most models can be split nicely.

For that you'd probably be better off removing one of the GPUs, and replacing it with a networking card.

The problem of the form factor will remain. The tinybox is 15U big for compute that you'd normally expect to find in a 4U form factor.

Re: "Tinyboxes finally have a buy it now button"

#83

Post by andersa »

StrangeDoctor wrote: post #78
andersa wrote: post #56

Earlier quoted context omitted.

You can't fine tune a 70B model with this. It barely even fits the weights before it runs out of vram. Need a bigger machine.

I think the idea is you network them together if you need more and most models can be split nicely.

Most models actually can't be split nicely by 6. There's a reason nvidia builds nodes with 4 and 8 GPUs.

Re: "Tinyboxes finally have a buy it now button"

#84

Post by caeril »

bob1029 wrote: post #66

I had a preorder in for this but I canceled a few weeks back. My experience trying to run machines this powerful in residential settings has been extremely poor. All of the Seasonic power supplies that go beyond 1kW or so will trip my shitty (i.e. probably defective) Siemens AFCI breakers. Not even the same circuit all the time. Even after violating local electrical code, I have found that living with a 1500w+ monste…

If you're spending $15k on a box, you can also spend $1200 for a small insulated shed kit and $800 for a small mini-split heat pump. I live in a much warmer summer climate than Texas and this solution works fine for me for my small network cabinet.

Re: "Tinyboxes finally have a buy it now button"

#85

Post by andersa »

anon389r58r58 wrote: post #82
StrangeDoctor wrote: post #78

Earlier quoted context omitted.

I think the idea is you network them together if you need more and most models can be split nicely.

For that you'd probably be better off removing one of the GPUs, and replacing it with a networking card. The problem of the form factor will remain. The tinybox is 15U big for compute that you'd normally expect to find in a 4U form factor.

I don't think they're intended for rack usage like that. More like for people to put under their desks... there would be no reason to build the giant case with fancy silent-ish cooling if you're going to put them next to your other jet engines.

Re: "Tinyboxes finally have a buy it now button"

#86

Post by anon389r58r58 »

andersa wrote: post #85
anon389r58r58 wrote: post #82

Earlier quoted context omitted.

For that you'd probably be better off removing one of the GPUs, and replacing it with a networking card. The problem of the form factor will remain. The tinybox is 15U big for compute that you'd normally expect to find in a 4U form factor.

I don't think they're intended for rack usage like that. More like for people to put under their desks... there would be no reason to build the giant case with fancy silent-ish cooling if you're going to put them next to your other jet engines.

Fully agree, and I think the tinybox is great if you put only one of them somewhere in your local office.

I just don't think it makes sense to connect multiple of them into a "cluster" to work with bigger models, as the networking bandwidth isn't good enough and you'd have to fit multiple of these big boxes into your local space. Then I might as well put up a rack in a separate room.

Re: "Tinyboxes finally have a buy it now button"

#87

Post by ofrzeta »

teruakohatu wrote: post #73
bob1029 wrote: post #66

I had a preorder in for this but I canceled a few weeks back. My experience trying to run machines this powerful in residential settings has been extremely poor. All of the Seasonic power supplies that go beyond 1kW or so will trip my shitty (i.e. probably defective) Siemens AFCI breakers. Not even the same circuit all the time. Even after violating local electrical code, I have found that living with a 1500w+ monste…

> All of the Seasonic power supplies that go beyond 1kW or so will trip my shitty (i.e. probably defective) Siemens AFCI breakers. Not even the same circuit all the time. I don’t know much about US electrical standards but aren’t your residential circuits rated for 1800w or 2400w? Here in New Zealand they are 2400w and people regularly plug in 2400w fan heaters without issue. > The HVAC solutions in most residences w…

Crazy. Our hair-dryer has 2kW.

Re: "Tinyboxes finally have a buy it now button"

#88

Post by georgehotz »

jasomill wrote: post #32
SushiHippie wrote: post #15

Earlier quoted context omitted.

> 6x PCIe 4.0 x16 (64 GB/s) Wikipedia [0] states that PCIe 4.0 x16 has a throughput of ~32GB/s, what does the (64 GB/s) indicate on the website, is this just a typo and you have 6x ~32GB/s or does it mean in total you can "only" expect a throughput of 64GB/s all lanes slots combined? If so, wouldn't you also be bottlenecked by the PCIe bandwidth (when moving data between CPU and GPU)? [0] https://en.wikipedia.org/wik…

Most EPYCs have 128 PCIe lanes, so I'd expect a full x16 link for all six GPUs. Pedantically, the combined bidirectional bandwidth of PCIe x16 is ~64 GB/s, as it's a full-duplex ~32 GB/s link, but that's an awfully misleading spec if this is the intent (akin to claiming Gigabit Ethernet is 2 Gb/sec).

It's the same way NVIDIA states bandwidth for PCIe and NVLink.

https://www.nvidia.com/en-us/data-center/h100/

Re: "Tinyboxes finally have a buy it now button"

#89

Post by georgehotz »

anon389r58r58 wrote: post #69
vessenes wrote: post #22

I thought to myself this morning: "boy, that $15k pricetage is tempting." Then I thought to myself "how many times have I downloaded a github repo only to hand-replace cuda with mps, and then tried to figure out if there's a version of xformers that will work this week with my m3?" and then I thought "boy, that $25k is tempting." (15k: Radeon / 25k: Nvidia). For those wondering, 3200W power, in residential / low-end…

The networking of the tinybox is woefully inadequate. I.e. it only has an OCP 3.0 interface which is unoccupied. If you can fit everything onto one tinybox, then you'll be good, if you cannot, then you'd be better off by having a more professional workstation solution like e.g. NVIDIA RTX cards which have more memory.

That OCP 3.0 card has the same link bandwidth as the GPUs, so you can scale out without much loss of all-reduce bandwidth. In practice, for all models except the largest, the ~16GB/s all-reduce is totally fine. You just need to make sure you can all-reduce all weights in your training step time.

Say you are training a 3B parameter model in BF16. That's 6GB of weights, as long as your step time is >=500ms you won't see a slowdown.

Re: "Tinyboxes finally have a buy it now button"

#90

Post by georgehotz »

jagrsw wrote: post #55
bguebert wrote: post #24

Earlier quoted context omitted.

On their site they say it can run at 220V 15A if you got that. https://docs.tinygrad.org/tinybox/

Just being nitpicky - I'm from the EU, but I think in the US, you can get either: 240V: Split-phase, this gives you 120V between each leg and neutral, and 240V across the two legs. 208V: The interphase in a 3-phase system. Might be still within tolerance of 220V :) HTH, ducking out :)

Specs on website updated, anywhere from 100-240V is fine
Post reply on HN