Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

171–180 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#171

Post by trebligdivad »

g413n wrote: post #161
trebligdivad wrote: post #159

Earlier quoted context omitted.

I assume your actual training is being done somewhere else? Did you try getting colocation space in the same datacentre as somewhere with the compute - it would have reduced your internet costs even further.

yeah the cost calculus is very different for gpus, it absolutely makes sense for us to be using cloud there. also hardly any datacenters can support the power density, esp in downtown sf

Yeh; one other thing - you list a separate management network as an optional - it's not optional! Under no circumstance must you expose the managemnt IPs of switches or the servers to the internet; they are, on average, about as secure as a drunk politician. Use a separate management net, make sure it's only securly accessed.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#172

Post by mx7zysuj4xew »

yread wrote: post #119

You could get pretty close to the cost 1$/TB/month using Hetzner's sx135 with 8x22TB so 140TB in raidz1 for 240 eur. Maybe you get a better rate if you rent 200 of them. Someone else takes care of a lot of risks and you can sleep well at night

You cannot use hetzner for anything serious.

They'd most likely claim abuse and delete your data wholesale without notice

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#173

Post by echelon »

Scramblejams wrote: post #169

Fun piece, thanks to the author. But for vicarious thrills like this, more pictures are always appreciated!

If the authors chime in, I'd like to ask what "Standard Intelligence PBC" does.

Is it a public benefit corp?

What are y'all building?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#174

Post by luhn »

nodja wrote: post #133
jimmytucson wrote: post #75

Just wanted to say, thanks for doing this! Now the old rant... I started my career when on-prem was the norm and remember so much trouble. When you have long-lived hardware, eventually, no matter how hard you try, you just start to treat it as a pet and state naturally accumulates. Then, as the hardware starts to be not good enough, you need to upgrade. There's an internal team that presents the "commodity" interface…

Yeah from memory on-prem was always cheaper, it just removed a lot of logistic obstacles and made everything convenient under one bill. IIRC the wisdom of the time cloud started becoming popular was to always be on-prem and use cloud to scale up when demand spiked. But over time temporarily scaling up became permanent, and devs became reliant on instantly spawning new machines for things other than spikes in demand a…

> IIRC the wisdom of the time cloud started becoming popular was to always be on-prem and use cloud to scale up when demand spiked.

I've heard that before but was never able to make sense of it. Overflowing into the cloud seems like a nightmare to manage, wouldn't overbuilding on-prem be cheaper than paying your infra team to straddle two environments?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#176

Post by Symbiote »

g413n wrote: post #152
mwambua wrote: post #132

Earlier quoted context omitted.

How did you arrive at the decision of not putting the GPU machines in the colo? Were the power costs going to be too high? Or do you just expect to need more physical access to the GPU machines vs the storage ones?

When I was working at sfcompute prior to this we saw multiple datacenters literally catch on fire bc the industry was not experienced with the power density of h100s. Our training chips just aren't a standard package in the way JBODs are.

Isn't the easy option to spread the computers out, i.e. not fill the rack, but only half of it?

A GPU cluster next to my servers has done this, presumably they couldn't have 64A in one rack so they've got 32A in two. (230V 3phase.)

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#177

Post by Symbiote »

pighive wrote: post #153

HDDs - are never one time costs. Do datacenters also offer ordering and replacing HDDs?

They offer replacing, yes, but normally expect you to order the new one. (Usually covered by a warranty, sent next business day.)

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#178

Post by antisthenes »

joering2 wrote: post #92
gordonhart wrote: post #44

Earlier quoted context omitted.

Backblaze reports an annual failure rate of 1.36% [0]. Since their cluster uses 2,400 drives, they would likely see ~32 failures a year (extra ~$4,000 annual capex, almost negligible). [0] https://www.backblaze.com/cloud-storage/resources/hard-drive...

Their rate will probably be higher since they are utilizing used drives. From the spec: 2,400 drives. Mostly 12TB used enterprise drives (3/4 SATA, 1/4 SAS). The JBOD DS4246s work for either.

Not necessarily, since disk failures are typically U-shaped.

Buying used drives eliminates the high rate of early failure (but does get you a bit closer to the 2nd part of the U-curve).

Typically most drives would become more obsolete before hitting the high failure rate of the right side of the U-curve from longevity (7+ years)

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#180

Post by Symbiote »

datadrivenangel wrote: post #59
g413n wrote: post #40

Earlier quoted context omitted.

someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph

Assuming that they end up hiring a full time ops person at 500k annually total costs (250k base for a data center wizard), then that's 42k extra a month, or ~$70k. Still 200k per month lower than their next best offering.

It's really not necessary.

I have four racks rather than ten, and less storage but more compute. All purchased new from HP with warranties.

Ordering each year takes a couple of days work. Racking that takes one or two.

Initial setup (seeing differences with a new generation of server etc and customizing Ubuntu autoinstallation) is done in a day.

So that's a week per year for setup.

If we are really unlucky, add another week for a strange failure. (This happened once in the 10 years I've been doing this, a CPU needed replacement by the HP engineer.)

I replaced a couple of drives in July, and a network fibre transceiver in May.

Post reply on HN