Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

21–30 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#21

Post by g413n »

cjaackie wrote: post #16
g413n wrote: post #2

No mention of disk failure rates? curious how it's holding up after a few months

They mentioned the cluster being used enterprise drives, I can see the desire to save money but agree, that is going to be one expensive mistake down the road. I should also note personally for home cluster use, I learned quickly that used drives didn’t seem to make sense. Too much performance variability.

in a datacenter context failure rates are just a remote-hands recurring cost so it's not too bad with front-loaders

e.g. have someone show up to the datacenter with a grocery list of slot indices and a cart of fresh drives every few months.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#23

Post by giancarlostoro »

nee1r wrote: post #17
leejaeho wrote: post #12

how long do you think it'll be before you fill all of it and have to build another cluster LOL

Already filled up and looking to possibly copy and paste :)

So, others have asked, and I'm curious myself are you sourcing the videos yourselves or third parties?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#25

Post by guywithahat »

cjaackie wrote: post #16
g413n wrote: post #2

No mention of disk failure rates? curious how it's holding up after a few months

They mentioned the cluster being used enterprise drives, I can see the desire to save money but agree, that is going to be one expensive mistake down the road. I should also note personally for home cluster use, I learned quickly that used drives didn’t seem to make sense. Too much performance variability.

Used drives make sense if maintaining your home server is a hobby. It's fun to diagnose and solve problem in home servers, and failing drives give me a reason to work on the server. (I'm only half-joking, it's kind of fun)

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#28

Post by jms55 »

cjaackie wrote: post #16
g413n wrote: post #2

No mention of disk failure rates? curious how it's holding up after a few months

They mentioned the cluster being used enterprise drives, I can see the desire to save money but agree, that is going to be one expensive mistake down the road. I should also note personally for home cluster use, I learned quickly that used drives didn’t seem to make sense. Too much performance variability.

If I remember correctly, most drives either:

1. Fail in the first X amount of time

2. Fail towards the end of their rated lifespan

So buying used drives doesn't seem like the worst idea to me. You've already filtered out the drivers that would fail early.

Disclaimer: I have no idea what I'm talking about

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#29

Post by p_ing »

nee1r wrote: post #13
miniman1337 wrote: post #9

Used Disks, No DR, not exactly a real shoot out.

True, though this is specifically for pretraining data (S3 wouldn't sell us used disk + no DR storage).

You're in a seismically active part of the world. Will the venture last in a total loss scenario?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#30

Post by stackskipton »

mschuster91 wrote: post #20

Shows how crazy cheap on prem can be. tips hat

Not included is overhead of dealing with maintenance. S3/R2 generally don’t require OPS type dedicated to care and feeding. This type of setup will likely require someone to spend 5 hours a week dealing with it.
Post reply on HN