Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

161–170 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#161

Post by g413n »

trebligdivad wrote: post #159
g413n wrote: post #147

Earlier quoted context omitted.

yeah misunderstanding we'll update the post-- separately it's true that we aren't network specialists and the network wrangling was prob disproportionately hard for us/ shouldn't have taken so long.

I assume your actual training is being done somewhere else? Did you try getting colocation space in the same datacentre as somewhere with the compute - it would have reduced your internet costs even further.

yeah the cost calculus is very different for gpus, it absolutely makes sense for us to be using cloud there. also hardly any datacenters can support the power density, esp in downtown sf

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#163

Post by XorNot »

trebligdivad wrote: post #97

The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…

I mean generally above a certain size of deployment DHCP is much more trouble then it's worth.

DHCP is really only worth it when your hosts are truly dynamic (i.e. not controlled by you). Otherwise it's a lot easier to handle IP allocation as part of the asset lifecycle process.

Heck even my house IoT network is all static IPs because at the small scale it's much more robust to not depend on my home router for address assignment - replacing a smart bulb is a big enough event, so DHCP is solely for bootstrapping in that case.

At the enterprise level unpacking a server and recording the asset IDs etc is the time to assign IP addresses.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#164

Post by g413n »

Havoc wrote: post #150

Cool write-up. I do feel sorry for the friends that go suckered into doing a bunch of grunt work for free though

yeah that's why we started paying people near the second half- not super clearly stated in the blogpost, but the novelty definitely wore off with plenty of drives left to stack, so we switched strategies to get it done in time.

I think everyone who showed up for a couple hours as part of the party had a good time tho, and the engraved hard drives we were giving out weren't cheap :p

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#167

Post by fuzzylightbulb »

nodja wrote: post #128
yread wrote: post #119

You could get pretty close to the cost 1$/TB/month using Hetzner's sx135 with 8x22TB so 140TB in raidz1 for 240 eur. Maybe you get a better rate if you rent 200 of them. Someone else takes care of a lot of risks and you can sleep well at night

I don't think Hetzner provides locations in SF. Those 100GBit connections don't do much if they need to connect outside the city the rest of the equipment is in, but maybe peering has gotten better and my views are outdated.

You're good. The speed of light through a glass fiber is still just as slow as it ever was.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#168

Post by matt-p »

jimmytucson wrote: post #75

Just wanted to say, thanks for doing this! Now the old rant... I started my career when on-prem was the norm and remember so much trouble. When you have long-lived hardware, eventually, no matter how hard you try, you just start to treat it as a pet and state naturally accumulates. Then, as the hardware starts to be not good enough, you need to upgrade. There's an internal team that presents the "commodity" interface…

Docker is amazing for forcing the machines not to be pets, seriously, a racked sever is just another K3 or K8 node (or whatever) and doesn't get the choice or ability of being petted. It's so nice. You could maybe of said the same about vm's but not really, the VM just became the pet, OK you could at least image/snapshot it but it's not the same.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#170

Post by Symbiote »

trebligdivad wrote: post #97

The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…

I have static, public IPs across 80 or so servers.

It gets set approximately once when the server's automated Ubuntu installation runs, and I never think about it.

> Where does the switch choice come into whether you DHCP?

Perhaps from home routers which include I've.

> Wth would you want public IPs.

Why wouldn't you? They have a firewall.

Post reply on HN