Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

81–90 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#81

Post by theideaofcoffee »

jimmytucson wrote: post #75

Just wanted to say, thanks for doing this! Now the old rant... I started my career when on-prem was the norm and remember so much trouble. When you have long-lived hardware, eventually, no matter how hard you try, you just start to treat it as a pet and state naturally accumulates. Then, as the hardware starts to be not good enough, you need to upgrade. There's an internal team that presents the "commodity" interface…

I'm not op, but thanks for this. Like I mentioned in another comment, the wholesale move to the cloud has caused so many skills to become atrophied. And it's good that someone is starting to exercise that skill again, like you said. The hyperscalers are mostly to blame for this, the marketing FUD being that you can't possibly do it yourself, there are too many things to keep track of, let us do it (while conveniently leaving out how eye-wateringly expensive they are in comparison).

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#82

Post by g413n »

jimmytucson wrote: post #75

Just wanted to say, thanks for doing this! Now the old rant... I started my career when on-prem was the norm and remember so much trouble. When you have long-lived hardware, eventually, no matter how hard you try, you just start to treat it as a pet and state naturally accumulates. Then, as the hardware starts to be not good enough, you need to upgrade. There's an internal team that presents the "commodity" interface…

we're in a pretty unique situation in that very early on we fundamentally can't afford the hyperscaler clouds to cover operations, so we're forced to develop some expertise. turned out to be reasonably chill and we'll prob stick with it for the foreseeable future, but we have seen a little bit of the state-creep you mention so tbd.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#83

Post by g413n »

jms55 wrote: post #28
cjaackie wrote: post #16

Earlier quoted context omitted.

They mentioned the cluster being used enterprise drives, I can see the desire to save money but agree, that is going to be one expensive mistake down the road. I should also note personally for home cluster use, I learned quickly that used drives didn’t seem to make sense. Too much performance variability.

If I remember correctly, most drives either: 1. Fail in the first X amount of time 2. Fail towards the end of their rated lifespan So buying used drives doesn't seem like the worst idea to me. You've already filtered out the drivers that would fail early. Disclaimer: I have no idea what I'm talking about

we don't have perfect metrics here but this seems to match our experience; a lot of failures happened shortly after install before the bulk of the data download onto the heap, so actual data loss is lower than hardware failure rates

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#84

Post by wongarsu »

Aurornis wrote: post #73
calvinmorrison wrote: post #70

Earlier quoted context omitted.

> The amount of time lost to driving to the datacenter, waiting for replacement parts to arrive, and scrambling to patch over unexpected failure modes is always much higher than expected. I don't have this experience at all. Our colo handled almost all work. the only time i ever went to the server farm was to build out whole new racks. Even replacing servers the colo handled for us at good cost. Our reliability came…

> Our reliability came from software not hardware, though of course we had hundreds of spares sitting by, the defense in depth (multiple datacenters, each datacenter having 2 'brains' which could hotswap, each client multiply backed up on 3-4 machines)... Of course, but building and managing the software stack, managing hundreds of spares across locations, spanning across datacenters, having a hotswap backup system i…

To be fair, what's described here is much more robust than what you get with a simple AWS setup. At a minimum that's a multi-region setup, but if the DCs have different owners I'd even compare it to a multi-cloud setup.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#85

Post by madsushi »

geor9e wrote: post #74
bayindirh wrote: post #7

Earlier quoted context omitted.

They can rent a dark fiber for themselves for that distance, and it'll be cheap. However, as they noted they use 100gbps capacity from their ISP.

Does San Francisco really still have dark fiber? That 90s bubble sure did overshoot demand.

DWDM tech improvements have outpaced nearly every other form of technology growth, so the same single pair of fiber that used to carry 10 Mbps can now carry 20 Tbps, which is a 2,000,000x multiplier. The same somewhat-fixed supply of fiber can go a very long way today, so the price pressure for access is less than you might expect.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#86

Post by g413n »

ttfvjktesd wrote: post #58
devanshp wrote: post #54

Earlier quoted context omitted.

we have 6 months of experience operating thousands of physical disks in datacenters now! it's about a couple hours a month of employee time in steady-state.

How about all the other infrastructure. Since you are obviously not using the cloud, you must have massive amounts of GPUs and operating systems. All of that has been working together, it's not just keep watching for the physical disks and all is set. Don't get me wrong, I buy the actual numbers regarding hardware costs, but in addition to that presenting the rest as basically a one man show in terms of maintenance h…

oh we use cloud gpus, infiniband h100s absolutely aren't something we want to self-host. not aws tho, they're crazy overpriced; mithril and sfcompute!

we also use cloudflare extensively for everything that isn't the core heap dataset, the convenience of buckets is totally worth it for most day-to-day usage.

the heap is really just the main pretraining corpus and nothing else.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#87

Post by g413n »

Sanzig wrote: post #62
nee1r wrote: post #38

Earlier quoted context omitted.

Definitely much less redundancy, this was definitely a tradeoff we made for pretraining data and cost.

Did you do any kind of redundancy at least (eg: putting every 10 disks in RAID 5 or RAID Z1)? Or I suppose your training application doesn't mind if you shed a few terabytes of data every so often?

atm we don't and we're a bit unsure whether it's a free lunch wrt adding complexity. there's a really nice property of having isolated hard drives where you can take any individual one and `sudo mount` it and you have a nice chunk of training data, and that's something anyone can feel comfortable touching without any onboarding to some software stack

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#88

Post by dylan604 »

g413n wrote: post #2

No mention of disk failure rates? curious how it's holding up after a few months

I've mentioned this story before, but we had massive drive failures when bringing up multiple disk arrays. We get them racked on a friday afternoon, and then I wrote a quick and dirty shell script to read/write data back and forth between them over the weekend that was to kick in after they finished striping the raid arrays. By quick and dirty I mean there was no logging, and just a bunch of commands saved as .sh. Came in on Monday to find massive failures in all of the arrays, but no insight into when they failed during the stripe or during stressing them. It was close to 50% failure rate. Turned out to be a bad batch from the factory. Multiple customers of our vendor were complaining. All the drives were replaced by the manufacturer. It just delayed the storage being available to production. After that, not one of them failed in the next 12 months before I left for another job.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#89

Post by g413n »

Sanzig wrote: post #66
nee1r wrote: post #13

Earlier quoted context omitted.

True, though this is specifically for pretraining data (S3 wouldn't sell us used disk + no DR storage).

I do appreciate the scrappiness of your solution. Used drives for a storage cluster is like /r/homelab on steroids. And since it's pretraining data, I suppose data integrity isn't critical. Most venture-backed startups would have just paid the AWS or Cloudflare tax. I certainly hope your VCs appreciate how efficient you are being with their capital :)

worth stressing that we literally could not afford pretraining without this, approx our entire seed round would go into cloud storage costs

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#90

Post by dylan604 »

nee1r wrote: post #11
bayindirh wrote: post #7

Earlier quoted context omitted.

They can rent a dark fiber for themselves for that distance, and it'll be cheap. However, as they noted they use 100gbps capacity from their ISP.

We want to get darkfiber from the datacenter to the office. I love 100Gbps

I'm now envisioning a poster with a strand of fiber wearing aviators with large font size Impact font reading Dark Fiber with literal laser beams coming out of the eyes.
Post reply on HN