Building the heap: racking 30 petabytes of hard drives for pretraining
1–10 of 281 posts
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#2No mention of disk failure rates? curious how it's holding up after a few months
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#3No mention of disk failure rates? curious how it's holding up after a few months
good point
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#4great write up, really appreciate the explanations / showing the process
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#5So how do they get this data to the GPUs now...? Just run it over the public internet to the datacenter?
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#6No mention of disk failure rates? curious how it's holding up after a few months
The disk failure rates are very low when compared to decade ago. I used to change more than a dozen disks every week a decade ago. Now it's an eyebrow raising event which I seldom see.
I think following Backblaze's hard disk stats is enough at this point.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#7So how do they get this data to the GPUs now...? Just run it over the public internet to the datacenter?
They can rent a dark fiber for themselves for that distance, and it'll be cheap.
However, as they noted they use 100gbps capacity from their ISP.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#8So how do they get this data to the GPUs now...? Just run it over the public internet to the datacenter?
7.5k for zayo 100gig so that's like half of the MRC
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#9Used Disks, No DR, not exactly a real shoot out.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#10So how do they get this data to the GPUs now...? Just run it over the public internet to the datacenter?
yeah, exactly! we have a 100G uplink, and then we use nginx secure links that we then just curl from the machines using HTTP. (funnily HTTPS adds overhead so we just pre-sign URLs)