Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building the heap: racking 30 petabytes of hard drives for pretraining

si.inc

Post reply on HN
Print view View original

231–240 of 281 posts

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#231

Post by znpy »

computably wrote: post #221
jillesvangurp wrote: post #216

AWS obviously does the same but better. That's why they are so rich. Their cost is a tiny percentage of their revenue. They buy cheap servers, and then run lots of vms on them. Each of which delivers 10s/100s of $ per month. That server pays for itself in revenue within weeks/months. And it will be in service until it stops working which could be over five years. Same with storage, networking, gpus, etc. They've spen…

> AWS at cost price would probably be 60-80x less than what they charge; if not more. If you look at https://ir.aboutamazon.com/news-release/news-release-details... , it says in 2024, > AWS segment sales increased 19% year-over-year to $107.6 billion. > AWS segment operating income was $39.8 billion, compared with operating income of $24.6 billion in 2023. So about 59% margin, relative to costs. Everybody undercuttin…

> AWS segment operating income was $39.8 billion

keep in mind that cost there is likely to also be including personnel. probably a significant fraction if you consider how many employees amazon's aws division has.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#232

Post by toast0 »

pixl97 wrote: post #186
Symbiote wrote: post #176

Earlier quoted context omitted.

Isn't the easy option to spread the computers out, i.e. not fill the rack, but only half of it? A GPU cluster next to my servers has done this, presumably they couldn't have 64A in one rack so they've got 32A in two. (230V 3phase.)

Rackspace is typically at a premium at most data centers.

My info may be dated, but power density has gone up a ton over time. I'd expect a lot of datacenters to have plenty of space, but not much power. You can only retrofit so much additional power distribution and cooling into a building designed for much less power density.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#233

Post by supermatt »

hmcamp wrote: post #162
supermatt wrote: post #157

Where does one get “90 million hours of video data”?

I’m also curious about this. I don’t recall seeing that mentioned in the article

Its in the first sentence: "We built a storage cluster in downtown SF to store 90 million hours worth of video data."

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#234

Post by toast0 »

antisthenes wrote: post #178
joering2 wrote: post #92

Earlier quoted context omitted.

Their rate will probably be higher since they are utilizing used drives. From the spec: 2,400 drives. Mostly 12TB used enterprise drives (3/4 SATA, 1/4 SAS). The JBOD DS4246s work for either.

Not necessarily, since disk failures are typically U-shaped. Buying used drives eliminates the high rate of early failure (but does get you a bit closer to the 2nd part of the U-curve). Typically most drives would become more obsolete before hitting the high failure rate of the right side of the U-curve from longevity (7+ years)

I bet you still have a higher early failure rate because of the stress from transportation, even if there's no funny business. And I expect some funny business because used enterprise drives often come with wiped SMART data, some drives may have been retired by sophisticated clients who decided they were near failure.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#235

Post by toast0 »

tarasglek wrote: post #101
OliverGuy wrote: post #99

Earlier quoted context omitted.

The NetApps are just disk shelves, can plug it into a SAS controller and use whatever software stack you please.

but they have multiple head nodes, so its some distributed setup or just active/passive type thing?

I think each rack is one head node and several disk shelves (10?). No dual headed shelves.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#236

Post by urbandw311er »

Well done! I love the honest write up and the “can do” attitude. Must have been a lot of fun too. Out of interest why do you think you made the mistake of buying 20x more drives than you needed instead of the denser storage that you mention? Was there a reason you opted for this?

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#237

Post by toast0 »

synack wrote: post #91

IPMI is great and all, but I still prefer serial ports and remote PDUs. Never met a BMC I could trust.

Serial over IPMI, plus ipmi power control is pretty good when it works. Supermicro X10 and newer was pretty nice. X9 and X8 not as nice; it's not helpful when the serial over ipmi drops during reboot and doesn't come back in a reasonable amount of time, and then the graphical mode needs ancient java webstart with os and platform specific jni, oof.

Re: Building the heap: racking 30 petabytes of hard drives for pretraining

#238

Post by immibis »

ranger_danger wrote: post #136
immibis wrote: post #61

Earlier quoted context omitted.

Very good point. There was something on the HN front page like this about self-hosted email, too. I point out to people that AWS is between ten to one hundred times more expensive than a normal server. The response is "but what if I only need it to handle peak load three hours a day?" Then you still come out ahead with your own server. We have multiple colo cages. We handle enough traffic - terabytes per second - tha…

I don't think the answer is so black-and-white. IMO This only realistically applies to larger companies or ones that either push lots of traffic or have a need for large amounts of compute/storage/etc. But for smaller groups that don't have large/sustained workloads, I think they can absolutely save money compared to colo/dedicated servers using one of multiple different kinds of AWS services. I have several customer…

Small companies should go for the likes of Hetzner/OVH, which is still 10+ times cheaper than AWS.

AWS is for anyone with a fear of committing to a particular amount of resource use, but once you've tried both and realised the price and performance differential, you realize you can easily way overcommit and still come out ahead, so it's not actually that scary. Plus, nobody's stopping you from continuing to spin up EC2s when your real servers are fully utilized.

Post reply on HN