271–280 of 374 posts
For online or nearline storage, you should look at what Backblaze did. Either buy hardware that is similar to what they did (basically disk shelves, you can cram ~100 drives into a 4U chassis) or if you are at that scale you can probably build your own just like they did.
What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…
> If you talk to Dell/HP/other, they can advise you and sell you large storage appliances. Problem is, the larger appliances will only host 1 or 2 PB. That's nowhere near enough. This is just incorrect. If you talk to HPE, they should be quite happy to sell you the my employer's software (Qumulo) alongside their hardware. 10+ PB is definitely supported. (The HPE part is not required) If you talk to Dell EMC, they wil…
Four years ago, one of the all flash vendors routinely advertised “well under a dollar a gigabyte”. Their prices have dropped dramatically since then, but the out of date numbers translate to “well under a million per PB”. That’s at the high end of performance with posix (nfs) or crash coherent (block) semantics. (Some also do S3, if that’s preferable for some reason)
With a 5 year depreciation cycle, those old machines were at Hope that helps.
At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…
I've rarely seen any non-giants with PBs of data properly compressed. For example, small JSON files converted into larger, compressed parquet files will use 10-100x less space. I am not familiar with images but see no reason why encoding batches of similar images should make it hard to get similar or even better compression ratios
Also, if you decide to move off later on, your transfer costs will also be cheaper if you can move it off in a compressed form first.
I would host in a datacenter of your choice and do a cross connect into AWS: https://aws.amazon.com/directconnect/pricing/ This allows you to read the data into AWS instances at no cost and process it as needed since there is 0 cost for ingress into AWS. I have some experience with this (hosting using Equinix)
Thanks for the pointer. Never thought about this as an option. Great stuff!!!
Hiring staff to build this would make sense at this point, because if your S3 storage cost is really $200,000/month, you can hire 3 good engineers with $450,000/year, which is the cost of just two months of S3 storage.
Earlier quoted context omitted.
> Let's do the math. Offers no math. At retail, 625 16TB drives is $400000. This is about 2x the MONTHLY retail s3 pricing. Further, as we all know, AWS bandwidth pricing is absolutely bonkers (1). I think your conclusion that S3 is "pretty good" needs a lot more math to support. (1) https://twitter.com/eastdakota/status/1371252709836263425
> 625 16TB drives is $400000 how much is the real estate cost of 625 drives and associated machinery to run it? At a guess, AWS has an operating margin of about 30%, so you can approximate their cost of hardware, bandwidth, and other fixed costs as 70% of their sticker price. As a start up, can you actually get this price to be lower? I actually dont think you can, unless your operation is very small and can be done…
> At a guess
The comments in this discussion that try to provide actual numbers show a fairly lopsided argument against S3. The comments that are advocating for S3 aren't as detailed.
You can look at this at the macro level, as on comment did, and see that one 1.2PB RackmountPro 4U server is $11K. Yes, of course you still need space and power. But at least this gives us actual numbers to play with as a base (e.g. buying 10 of these is less than what you'll spend on S3 in a month)
At a miro-level. You can spend $650 on a 16TB hard drive, or $650 on 16TB for 2 months of S3. Now, S3 is battle-tested, has redundancy, has power, has a cpu, has a network card (but not bandwidth), and is managed - unquestionable HUGE wins. But the hard drive (and other equipment) come with a 3-5 year warranty. Now, the difference between $650 for the hard drive, and $12000 for S3 over 3 years, won't let you: get the power, rent the racks, hire the staff, and invest in learning ceph. But the difference between $400K and $5million will.
What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…
> If you talk to Dell/HP/other, they can advise you and sell you large storage appliances. Problem is, the larger appliances will only host 1 or 2 PB. That's nowhere near enough. This is just incorrect. If you talk to HPE, they should be quite happy to sell you the my employer's software (Qumulo) alongside their hardware. 10+ PB is definitely supported. (The HPE part is not required) If you talk to Dell EMC, they wil…
Kinda disappointed the file solutions seem more complicated and nothing more simple to setup like some of the new databases are like CockroachDB or MongoDB are to use. I feel like reinventing the wheel is kinda bad as rather let people who are more experts in this field handle this stuff, but I hate the idea of vendor lock-in and forced to use other peoples servers, self hosting be nice from a single node to test to a cluster spanning multiple datacenters. Maybe there's a solution out there, I done some searching and just seems to go in circles. I seen one system but if you wanted to add or remove nodes in the future, you couldn't just "drain" a chunk server by moving it data.