Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

181–190 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#181

Post by [deleted] »

matt-p wrote: post #168
jimmytucson wrote: post #75

Just wanted to say, thanks for doing this! Now the old rant... I started my career when on-prem was the norm and remember so much trouble. When you have long-lived hardware, eventually, no matter how hard you try, you just start to treat it as a pet and state naturally accumulates. Then, as the hardware starts to be not good enough, you need to upgrade. There's an internal team that presents the "commodity" interface…

Docker is amazing for forcing the machines not to be pets, seriously, a racked sever is just another K3 or K8 node (or whatever) and doesn't get the choice or ability of being petted. It's so nice. You could maybe of said the same about vm's but not really, the VM just became the pet, OK you could at least image/snapshot it but it's not the same.

[deleted]

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#182

Post by lucb1e »

The linked Discord post is also interesting and fun to read. Most of the post is more serious but this is one of the small gems:

> One thing we discovered very quickly was that [world cup] goals scored showed up in our monitoring graphs. This was very cool because not only is it neat to see real-world events show up in your systems, but this gave our team an excuse to watch soccer during meetings. We weren’t “watching soccer during meetings”, we were “proactively monitoring our systems’ performance.”

https://discord.com/blog/how-discord-stores-trillions-of-mes...

It is linked as evidence for Discord using "less than a petabyte" of storage for messages. My best guess is that they multiplied node size and count from this post, which comes out to 708 TB for the old cluster and 648 in the new setup (presumably it also has some space to grow)

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#183

Post by doublerabbit »

matt-p wrote: post #168
jimmytucson wrote: post #75

Just wanted to say, thanks for doing this! Now the old rant... I started my career when on-prem was the norm and remember so much trouble. When you have long-lived hardware, eventually, no matter how hard you try, you just start to treat it as a pet and state naturally accumulates. Then, as the hardware starts to be not good enough, you need to upgrade. There's an internal team that presents the "commodity" interface…

Docker is amazing for forcing the machines not to be pets, seriously, a racked sever is just another K3 or K8 node (or whatever) and doesn't get the choice or ability of being petted. It's so nice. You could maybe of said the same about vm's but not really, the VM just became the pet, OK you could at least image/snapshot it but it's not the same.

I've found docker is as of a monstrous pet.

Docker is a monster that you have to treat as a pet. You've still got to pet it through stages of updating, monitoring, snapshots and networking. When the internal system breaks it's no different to a server collapsing.

Snapshots are a haircut for the monster, useful but can make things worse.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#184

Post by Symbiote »

nee1r wrote: post #63
huxley_marvit wrote: post #46

damn this is cool as hell. estimate on the maintenance cost in person-hours/month?

Around 2-5 hours/month, mostly powercycling the servers and replacing hard drives

You should be able to power cycle the servers from their management interfaces.

(But I have the luxury of everything being bought new from HP, so the interfaces are similar.)

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#185

Post by [deleted] »

jimmytucson wrote: post #75

Just wanted to say, thanks for doing this! Now the old rant... I started my career when on-prem was the norm and remember so much trouble. When you have long-lived hardware, eventually, no matter how hard you try, you just start to treat it as a pet and state naturally accumulates. Then, as the hardware starts to be not good enough, you need to upgrade. There's an internal team that presents the "commodity" interface…

[deleted]

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#186

Post by pixl97 »

Symbiote wrote: post #176
g413n wrote: post #152

Earlier quoted context omitted.

When I was working at sfcompute prior to this we saw multiple datacenters literally catch on fire bc the industry was not experienced with the power density of h100s. Our training chips just aren't a standard package in the way JBODs are.

Isn't the easy option to spread the computers out, i.e. not fill the rack, but only half of it? A GPU cluster next to my servers has done this, presumably they couldn't have 64A in one rack so they've got 32A in two. (230V 3phase.)

Rackspace is typically at a premium at most data centers.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#187

Post by matt-p »

g413n wrote: post #147
matt-p wrote: post #127

Earlier quoted context omitted.

This was not written by a network person, quite clearly. Hopefully it's just a misunderstanding, otherwise they do need someone with literally any clue about networks.

yeah misunderstanding we'll update the post-- separately it's true that we aren't network specialists and the network wrangling was prob disproportionately hard for us/ shouldn't have taken so long.

Massive props for getting it done anyway. For others reading: In general a switch should never run DHCPd, but will normally/often relay it for you, your arista's would 100% have supported relaying, but in this case it sounds like it might even be flat L2. Normally you'd host dhcpd on a server.

Some general feedback incase it's helpful.. -20K on contractors seems insane if we're talking about rack and stack for 10 racks. Many datacentres can be persuaded to do it for free as part of you agreeing to sign their contract. Your contractors should at least be using a server lift of some kind, again often provided kindly by the facility. If this included paying for server configuration and so on, then ignore that comment (bargin!).

-I would almost never expect to actually pay a setup fee (beyond something nominal like 500 per rack) to the datacentre either, certainly if you're going to be paying that fee it had better include rack and stack.

-A crash cart should not be used for a install of this size, the servers should be plugged into the network, and then automatically configured by a script/IPXE. It might sound intimidating or hard but it's not, doesn't even require IMPI (though frankly I would strongly, strongly recommend it, if you do't already have it). I would use managed switches for the management network too, for sure.

-Consider two switches, especially if they are second hand. The cost of the cluster not being usable for a few days while you source and install a replacement even here probably is still thousands.

-Personally not a big fan of the whole JBOD architecture and would have just filled by boots with single socket 4u supermicro chasis. To each their own, but JBOD's main benefit is a very small financial saving at the cost of quite a lot of drawbacks IMO. YMMV.

-Depending on who you use for GPUs, getting a private link or 'peering' to them might save you some cost and provide higher capacity.

-I'm kind of shocked that FMT2 didn't turn out much cheaper than your current colo, would expect less than those figures possibly with the 100G DIA included (normally about $3000/month no setup).

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#188

Post by twoodfin »

drnick1 wrote: post #105

Everyone should give AWS the middle finger and start doing this. Beyond cost, it's a matter of sovereignty over one's computing and data.

If this is a real market, I’d expect AWS to introduce S3 Junkyard with a similar durability and cost structure.

They probably still won’t budge on the egress fees.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#189

Post by landryraccoon »

datadrivenangel wrote: post #140
landryraccoon wrote: post #134

Their electricity costs are $10K per month or about $120K per year. At an interest rate of 7% that's $1.7M of capital tied up in power bills. At that rate I wonder if it makes sense to do a massive solar panel and battery installation. They're already hosting all of their compute and storage on prem, so why not bring electricity generation on prem as well?

At 120K per year over the three year accounting life of the hardware, that's 360k... how do you get to 1.7M?

It seems unlikely to me that they'll never have to retrain their model to account for new data. Is the assumption that their power usage drastically drops after 3 years?

Unless they go out of business in 3 years that seems unlikely to me. Is this a one-off model where they train once and it never needs to be updated?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#190

Post by Symbiote »

trebligdivad wrote: post #171
g413n wrote: post #161

Earlier quoted context omitted.

yeah the cost calculus is very different for gpus, it absolutely makes sense for us to be using cloud there. also hardly any datacenters can support the power density, esp in downtown sf

Yeh; one other thing - you list a separate management network as an optional - it's not optional! Under no circumstance must you expose the managemnt IPs of switches or the servers to the internet; they are, on average, about as secure as a drunk politician. Use a separate management net, make sure it's only securly accessed.

I understood that it's optional because they can walk down the road to the data center instead.

They mention plugging monitors in several times. I think I've only done that once in the last couple of years, when a firmware upgrade failed and reset the management interface IP.

Post reply on HN