I have worked in HPC (academia) where the cluster storage size is measured in multiples of PB since a decade. Since latency and bandwidth is a killer requirement there, Infiniband (instead of Ethernet) is the defacto standard for connecting the storage pools to the computing nodes. Maintaining such a (storage) cluster requires 1-2 people on site which replace a few hard disks every day. Nevertheless, when I would con…
Ask HN: How would you store 10PB of data for your startup today?
121–130 of 374 posts
Re: Ask HN: How would you store 10PB of data for your startup today?
#122That said, if you really think you must, spend effort on good deduping/transcoding (relatively easy with images/video), and consider some far lower-cost storage options than S3, which is pretty pricey no matter what you do. If S3 is a good fit, I hear good things about Wasabi, but haven't used it myself.
If you have the technical ability (non-trivial, you need someone who really understands, disk and system I/O, RAID Controllers, PCI lane optimization, SAN protocols and network performance (not just IP), etc.) and the wherewithal to invest, then putting this on good hardware with something like say, ZFS at your site or a good co-lo will be WAY cheaper and probably offer higher performance than any other option, especially combined with serious deduping. (Look carefully at everything that comes in once and you never have to do it again.) Also, keep in mind that even-numbered RAID levels can make more sense for video streaming, if that's a big part of the mix.
The MAIN thing: Keep in mind that understanding your data flows is way more important than just "designing for scale". And really try to not need so much data in the first place.
(Aside: I'm was cofounder and chief technologist of one of the first onsite storage service providers - we built a screamer of a storage system that was 3-4x as fast, and scaled 10x larger than IBM's fastest Shark array, at less than 10% of the cost. The bad news - we were planning to launch the week of 9/11 and, as self-funded, ran out of money before the economy came back. The system kicked ass, though.)
Re: Ask HN: How would you store 10PB of data for your startup today?
#123It seems to me like you could save a ton of money by using your own hardware. Perhaps buy a bunch of big Synology boxes? At that scale you should also consider looking at technologies such as Ceph. We've recently switched to a setup with several Synology boxes for around 1PB net storage.
Funny you should mention this. I once worked at a startup that stored lots of remote sensing data. Their strategy was to put it on a Synology. When the Synology filled up, they bought another, and so forth. Only some of the Synologys were online at any particular time, and there was no indexing to find which Synology held what data. Plus, there were no backups so if one Synology were to blow up, all the data on it wa…
Re: Ask HN: How would you store 10PB of data for your startup today?
#124Earlier quoted context omitted.
The is amazing. Thank you. I’ve been looking at Backblaze storage pods that seem to be designed for that use case. Never rented rack space. Do you remember somehow the math on how much cheaper it was or how you thought about upfront cost vs ongoing. Just order of magnitude would be great.
Roughly a decade ago S3 storage pricing had a ~10x premium over self-hosted. The convenience of not having to touch any hardware is expensive.
Your colo facility will almost certainly have 24/7 staff on hand who can help you with tasks like swapping disks from a pile of spares, but expect to pay $300+ minimum just to get someone to walk over to your racks, even if the job is 10 mins.
With that said, the cost savings can still be enormous. But know what you're getting into.
Re: Ask HN: How would you store 10PB of data for your startup today?
#125At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…
Re: Ask HN: How would you store 10PB of data for your startup today?
#12610PB is a crazy amount of data. Far more than any normal business would ever have to deal with. Presuming you aren't crazy, you must have an unusual business plan to legitimately need to handle that much data. That means it's tough for us to say much - any assumptions we might have about it could be invalid depending on your actual business needs. You're just going to have to tell us some more about your business cas…
Re: Ask HN: How would you store 10PB of data for your startup today?
#127RAM?
Re: Ask HN: How would you store 10PB of data for your startup today?
#128At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…
Does Snowball let you exfiltrate data from AWS? I was under the impression it was only for bulk ingestion.
Re: Ask HN: How would you store 10PB of data for your startup today?
#129Wasabi is a good option. They’re S3 compatible and don’t charge any egress or ingress fees. Been using them for a few years. Great speeds and customer support.
Re: Ask HN: How would you store 10PB of data for your startup today?
#130(I'm not working there anymore, posting this just to help)