Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

41–50 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#41

Shows how crazy cheap on prem can be. tips hat

Not included is overhead of dealing with maintenance. S3/R2 generally don’t require OPS type dedicated to care and feeding. This type of setup will likely require someone to spend 5 hours a week dealing with it.

Why 5h a week? Just for hardware?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#42
post #29
post #13

Earlier quoted context omitted.

True, though this is specifically for pretraining data (S3 wouldn't sell us used disk + no DR storage).

You're in a seismically active part of the world. Will the venture last in a total loss scenario?

They spent $300,000 on drives, with AWS they would have spent 4x that PER MONTH. They're already ahead of the cloud.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#44
post #2

No mention of disk failure rates? curious how it's holding up after a few months

The disk failure rates are very low when compared to decade ago. I used to change more than a dozen disks every week a decade ago. Now it's an eyebrow raising event which I seldom see. I think following Backblaze's hard disk stats is enough at this point.

Backblaze reports an annual failure rate of 1.36% [0]. Since their cluster uses 2,400 drives, they would likely see ~32 failures a year (extra ~$4,000 annual capex, almost negligible).

[0] https://www.backblaze.com/cloud-storage/resources/hard-drive...

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#45
It's quite cheap to just store data at rest, but I'm pretty confused by the training and networking set up here. It sounds like from other comments that you're not going to put the GPUs in the same location, so you'll be doing all training over X 100 Gbps lines between sites? Aren't you going to end up totally bottlenecked during pretraining here?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#47

Earlier quoted context omitted.

Not included is overhead of dealing with maintenance. S3/R2 generally don’t require OPS type dedicated to care and feeding. This type of setup will likely require someone to spend 5 hours a week dealing with it.

I once had about three racks full of servers under my control, admittedly they weren't a ton of disks, but still the hardware maintenance effort was pretty much negligible over a few years (until it all went to the cloud). The majority of server wrangling work I spent dealing with OS updates and, most annoyingly, OpenStack. But that's something you can't escape even if you run your stuff in the cloud...

With S3/R2 whatever, you do get away from it. You dump a bunch of files on them and then retrieve them. OS Updates, Disk Failures, OpenStack, additional hardware? Pssh, that's S3 company problem, not yours.

$LastJob we ran a ton of Azure Web App Containers, alot of OS work no longer existed so it's possible with Cloud to remove alot of OS toil.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#48
post #40

The biggest part that is always missing in such comparisons is the employee salaries. In the calculation they give $354k/year of total cost per year. But now add the cost of staff in SF to operate that thing.

someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph

So the drives are never going to fail? PSUs are never going to burn out? You are never going to need to procure new parts? Negotiate with vendors?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#49
Nice writeup. All of the technical detail is great!

I'm curious about the process of getting colo space. Did you use a broker? Did you negotiate, and if so, how large was the difference in price between what you initially were quoted and what you ended up paying?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#50
post #40

The biggest part that is always missing in such comparisons is the employee salaries. In the calculation they give $354k/year of total cost per year. But now add the cost of staff in SF to operate that thing.

someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph

You are under the assumption that only Ceph (and similar complex software) requires staff, whereas plain 30 PB can be operated basically just by rebooting from time to time.

I think that anyone with actual experience of operating thousands of physical disks in datacenters would challenge this assumption.

Post reply on HN