IPMI is great and all, but I still prefer serial ports and remote PDUs. Never met a BMC I could trust.
Building the heap: racking 30 petabytes of hard drives for pretraining
91–100 of 281 posts
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#92Earlier quoted context omitted.
The disk failure rates are very low when compared to decade ago. I used to change more than a dozen disks every week a decade ago. Now it's an eyebrow raising event which I seldom see. I think following Backblaze's hard disk stats is enough at this point.
Backblaze reports an annual failure rate of 1.36% [0]. Since their cluster uses 2,400 drives, they would likely see ~32 failures a year (extra ~$4,000 annual capex, almost negligible). [0] https://www.backblaze.com/cloud-storage/resources/hard-drive...
Their rate will probably be higher since they are utilizing used drives. From the spec:
2,400 drives. Mostly 12TB used enterprise drives (3/4 SATA, 1/4 SAS). The JBOD DS4246s work for either.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#93IPMI is great and all, but I still prefer serial ports and remote PDUs. Never met a BMC I could trust.
Try Lenovo. Their BMCs Don't Suck (tm).
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#94Earlier quoted context omitted.
The disk failure rates are very low when compared to decade ago. I used to change more than a dozen disks every week a decade ago. Now it's an eyebrow raising event which I seldom see. I think following Backblaze's hard disk stats is enough at this point.
Backblaze reports an annual failure rate of 1.36% [0]. Since their cluster uses 2,400 drives, they would likely see ~32 failures a year (extra ~$4,000 annual capex, almost negligible). [0] https://www.backblaze.com/cloud-storage/resources/hard-drive...
[deleted]
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#95My question isn't why do it yourself. A quick back of the envelope math shows AWS being much more expensive. My question is why San Francisco? It's one of the most expensive real estate markets in the US (#2 residential, #1 commercial), and electricity is expensive. $0.71/KwH peak residential rate! A jaunt down 280 to San Jose's gonna be cheaper, at the expense of. having to take that drive to get hands on. But I'm sure you can find someone who's capable of running a DC that lives in San Jose and needs a job so the SF team doesn't have to commute down to South Bay. Now obviously there's something to be said for having the rack in the office, I know of at least two (three, now) in San Francisco, it just seems like a weird decision if you're already worrying about money to the point of not using AWS.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#96i am still confused what their software stack is, they dont use ceph but bought netapp, so they use nfs?
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#97The networking stuff seems....odd.
'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.'
Where does the switch choice come into whether you DHCP? Wth would you want public IPs.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#98Aren't those netapp shelves pretty old at this point? See a lot of people recommending against them even for homelab type uses. You can get those 60 drive SuperMicro JBODs for pretty cheap now, and those aren't too old, would have been my choice.
Plus, the TCO is already way under the cloud equiv. so might as well spend a little more to get something much newer and more reliable
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#99i am still confused what their software stack is, they dont use ceph but bought netapp, so they use nfs?
The NetApps are just disk shelves, can plug it into a SAS controller and use whatever software stack you please.
Re: Building the heap: racking 30 petabytes of hard drives for pretraining
#100The networking stuff seems....odd. 'Networking was a substantial cost and required experimentation. We did not use DHCP as most enterprise switches don’t support it and we wanted public IPs for the nodes for convenient and performant access from our servers. While this is an area where we would have saved time with a cloud solution, we had our networking up within days and kinks ironed out within ~3 weeks.' Where doe…
> Wth would you want public IPs.
So anyone can download 30 PB of data with ease of course.