Gusterfs + ZFS
Ask HN: How would you store 10PB of data for your startup today?
341–350 of 374 posts
Re: Ask HN: How would you store 10PB of data for your startup today?
#342Perhaps look into Vast Data? They have a TCO calculator [1] but it seems to compare to other on-prem data storage providers (like Isolon...). 10PB in One Zone IA costs $100,000/mo without discount, or $1.2M per year, and that's just for storage alone. Vast claims something like $3.5M TCO over 5 years with 10PB of data and no growth assumption. 5 years on your S3 zone with no data growth (or transfer...) is $6M.
Re: Ask HN: How would you store 10PB of data for your startup today?
#343Just run another venture round and dont think too hard about this problem. If everything goes well it wont be your problem for much longer, if it goes bad then who cares anyway.
Re: Ask HN: How would you store 10PB of data for your startup today?
#344Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed servic…
> The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph, I think at that scale you would want a ceph expert on staff as a full time salaried position. For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it. > EDIT: btw,…
OK, clarification: most of our fleet has a LOT of supermicro machines, where it's impossible to identify the drive unless by serial number. There's on UID light, the machines needs to go offline, plus some 10 screws needs to come out to open the chassis, 4 more by drives.
The amount of downtime this would generate for a single machine plus the operation cost doesn't worth the hassle unless the machine loses a significant chunk of drives.
Re: Ask HN: How would you store 10PB of data for your startup today?
#345With a volume like that you should negotiate at least three storage+CDN providers and see who will give you the best offer. It could be as much as 50% off street price and even more if you are ready to sign a 2-3 years contract.
I personally would consider S3 Glacier+CloudFront, member of Bandwidth Alliance [0] of your choice+CloudFlare, and whomever serves TikTok now.
Re: Ask HN: How would you store 10PB of data for your startup today?
#346At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…
You didn't take personnel cost into account. You will need at least two system administrators to look after those racks (even if remote hands to change faulty drives are in the monthly opex). It quickly takes you to surplus of 200k/year with current prices (which will rise another 50% in 5 years).
On the other hand, you may negotiate a very sizable discount from AWS for 10Pb storage for 5 years.
Re: Ask HN: How would you store 10PB of data for your startup today?
#347Earlier quoted context omitted.
> The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph, I think at that scale you would want a ceph expert on staff as a full time salaried position. For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it. > EDIT: btw,…
OK, clarification: most of our fleet has a LOT of supermicro machines, where it's impossible to identify the drive unless by serial number. There's on UID light, the machines needs to go offline, plus some 10 screws needs to come out to open the chassis, 4 more by drives. The amount of downtime this would generate for a single machine plus the operation cost doesn't worth the hassle unless the machine loses a signifi…
Chassis built for mass storage usually have lever caddies backed on to hotplug SAS backplanes.
Re: Ask HN: How would you store 10PB of data for your startup today?
#348Earlier quoted context omitted.
couple be wrong but I don't believe compression of batches of compressed images compresses well but it'd be very interested to here about techniques on this because I have a lot of space eaten up by timelapses myself
On the contrary, batches of images with a high degree of similarity compress _very_ well. You have to use an algorithm specifically designed for that task though. Video codecs are a real world example of such - consider that H. 265 is really compressing a stream of (potentially) completely independent frames under the hood. I'm not sure what the state of lossless algorithms might be for that though.
A movie is a series of similar images and while it does allow temporal compression in a 3rd axis to the 2d raster, H265 is about as good as it gets at the moment but its also lossy which might not be tolerable.
Re: Ask HN: How would you store 10PB of data for your startup today?
#349At that scale I would contact AWS, Backblaze and Wasabi directly to see what improvements they can offer in terms of TCO (and potentially for a longer term contract).
+1
and tell each that you contact others. Lowest bidder wins.
Re: Ask HN: How would you store 10PB of data for your startup today?
#350Not an experience. But if I was given the task, I'll probably think about how those data could be distributed. Maybe use my own instance of IPFS, so each 'node' don't have to store all of the data.