Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

261–270 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#261

Post by ranger_danger »

immibis wrote: post #238
ranger_danger wrote: post #136

Earlier quoted context omitted.

I don't think the answer is so black-and-white. IMO This only realistically applies to larger companies or ones that either push lots of traffic or have a need for large amounts of compute/storage/etc. But for smaller groups that don't have large/sustained workloads, I think they can absolutely save money compared to colo/dedicated servers using one of multiple different kinds of AWS services. I have several customer…

Small companies should go for the likes of Hetzner/OVH, which is still 10+ times cheaper than AWS. AWS is for anyone with a fear of committing to a particular amount of resource use, but once you've tried both and realised the price and performance differential, you realize you can easily way overcommit and still come out ahead, so it's not actually that scary. Plus, nobody's stopping you from continuing to spin up E…

Hard disagree... I think these black-and-white opinions are disingenuous, lack important nuance and are often just incorrect.

I even have customers on $3/mo EC2 instances... the cheapest dedicated server on OVH is still twenty times more expensive than that. I don't think there's any way to "come out on top" with OVH in that scenario, short of maybe claiming that the customer is somehow "doing it wrong" by only paying for what they need.

And yes hetzner/ovh have $3-4 cloud instances too, but now you're just directly competing with AWS and I don't see any benefit to call one any better than the other.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#262

Post by rtp4me »

Aurornis wrote: post #257
rtp4me wrote: post #250

Earlier quoted context omitted.

For drive issues, this is easy. Have a stack of replacements on hand and just open a "remote-hands" ticket with the CoLo provider to swap out the drive. This can usually be done in 1-2hrs from opening the ticket. For server issues; again, pretty easy. Just use iKVM/IPMI and iPXE to diagnose a faulty server. Again, using "remote-hands" from the CoLo provider can help fix problems if your staff does not have the skills…

In my experience, the issues that take 80% of your time are the unexpected edge cases, not the easy fixes. Swapping drives is basically the easiest fix. The issues that cause the most problems are the hard to diagnose ones like the faulty RAM that flips a bit every once in a while or the hard drive controller that triggers an driver bug with weird behavior that doesn’t show up in the logs with anything meaningful.

Sure, but realistically, how often does this really happen? I have probably replaced 3 or 4 DIMMs over the past few years. Hardware is very reliable these days.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#263

Post by supermatt »

amarcheschi wrote: post #256
supermatt wrote: post #233

Earlier quoted context omitted.

Its in the first sentence: "We built a storage cluster in downtown SF to store 90 million hours worth of video data."

They were asking for the source of those data

/facepalm

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#264

Post by fapjacks »

mx7zysuj4xew wrote: post #172
yread wrote: post #119

You could get pretty close to the cost 1$/TB/month using Hetzner's sx135 with 8x22TB so 140TB in raidz1 for 240 eur. Maybe you get a better rate if you rent 200 of them. Someone else takes care of a lot of risks and you can sleep well at night

You cannot use hetzner for anything serious. They'd most likely claim abuse and delete your data wholesale without notice

100% this. Hetzner has no problems completely blowing away whatever you've got running for arbitrary reasons. And their support is incresibly bad.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#265

Post by g413n »

lucb1e wrote: post #182

The linked Discord post is also interesting and fun to read. Most of the post is more serious but this is one of the small gems: > One thing we discovered very quickly was that [world cup] goals scored showed up in our monitoring graphs. This was very cool because not only is it neat to see real-world events show up in your systems, but this gave our team an excuse to watch soccer during meetings. We weren’t “watchin…

yeah we weren't sure about putting that number esp whether it includes all the image attachments, but in any case it's at least around the right reference class for the largest text data operations.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#266

Post by g413n »

urbandw311er wrote: post #236

Well done! I love the honest write up and the “can do” attitude. Must have been a lot of fun too. Out of interest why do you think you made the mistake of buying 20x more drives than you needed instead of the denser storage that you mention? Was there a reason you opted for this?

I think <2x more drives than needed, not 20x (24 vs 14TB), but the racks holding the drives could've been denser. Around the same cost in any case and our colo doesn't charge for space, so it's not a big deal and we were just going with what we were familiar with, but something to try.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#267

Post by g413n »

toast0 wrote: post #232
pixl97 wrote: post #186

Earlier quoted context omitted.

Rackspace is typically at a premium at most data centers.

My info may be dated, but power density has gone up a ton over time. I'd expect a lot of datacenters to have plenty of space, but not much power. You can only retrofit so much additional power distribution and cooling into a building designed for much less power density.

yep this was the case for us.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#268

Post by g413n »

twoodfin wrote: post #188
drnick1 wrote: post #105

Everyone should give AWS the middle finger and start doing this. Beyond cost, it's a matter of sovereignty over one's computing and data.

If this is a real market, I’d expect AWS to introduce S3 Junkyard with a similar durability and cost structure. They probably still won’t budge on the egress fees.

we would be so down to buy s3 junkyard tbh we were going around begging various storage clouds to offer us this before giving up and building it ourselves

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#269

Post by g413n »

Symbiote wrote: post #190
trebligdivad wrote: post #171

Earlier quoted context omitted.

Yeh; one other thing - you list a separate management network as an optional - it's not optional! Under no circumstance must you expose the managemnt IPs of switches or the servers to the internet; they are, on average, about as secure as a drunk politician. Use a separate management net, make sure it's only securly accessed.

I understood that it's optional because they can walk down the road to the data center instead. They mention plugging monitors in several times. I think I've only done that once in the last couple of years, when a firmware upgrade failed and reset the management interface IP.

yep this. we just turned off management
Post reply on HN