Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

121–130 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#121
post #84

I have worked in HPC (academia) where the cluster storage size is measured in multiples of PB since a decade. Since latency and bandwidth is a killer requirement there, Infiniband (instead of Ethernet) is the defacto standard for connecting the storage pools to the computing nodes. Maintaining such a (storage) cluster requires 1-2 people on site which replace a few hard disks every day. Nevertheless, when I would con…

Since he needs 1000ms response on storage isn't ethernet the better option? It can reach 400gb/s on fastest hardware now. I thought Infiniband was only reasonable to use when machines need to quickly access other machines primary memory. I would like to know if I'm wrong about this though.

Re: Ask HN: How would you store 10PB of data for your startup today?

#122
Actual answer: There is almost NO company that really needs that much data. This has mostly just become a pissing match. In general, companies (especially startups) are way better off making sure they have a small amount of high-quality, accurate, data than a huge pile-o-dung that they think they're going to use magical AI/ML pixie dust to do something with.

That said, if you really think you must, spend effort on good deduping/transcoding (relatively easy with images/video), and consider some far lower-cost storage options than S3, which is pretty pricey no matter what you do. If S3 is a good fit, I hear good things about Wasabi, but haven't used it myself.

If you have the technical ability (non-trivial, you need someone who really understands, disk and system I/O, RAID Controllers, PCI lane optimization, SAN protocols and network performance (not just IP), etc.) and the wherewithal to invest, then putting this on good hardware with something like say, ZFS at your site or a good co-lo will be WAY cheaper and probably offer higher performance than any other option, especially combined with serious deduping. (Look carefully at everything that comes in once and you never have to do it again.) Also, keep in mind that even-numbered RAID levels can make more sense for video streaming, if that's a big part of the mix.

The MAIN thing: Keep in mind that understanding your data flows is way more important than just "designing for scale". And really try to not need so much data in the first place.

(Aside: I'm was cofounder and chief technologist of one of the first onsite storage service providers - we built a screamer of a storage system that was 3-4x as fast, and scaled 10x larger than IBM's fastest Shark array, at less than 10% of the cost. The bad news - we were planning to launch the week of 9/11 and, as self-funded, ran out of money before the economy came back. The system kicked ass, though.)

Re: Ask HN: How would you store 10PB of data for your startup today?

#123
post #72
post #56

It seems to me like you could save a ton of money by using your own hardware. Perhaps buy a bunch of big Synology boxes? At that scale you should also consider looking at technologies such as Ceph. We've recently switched to a setup with several Synology boxes for around 1PB net storage.

Funny you should mention this. I once worked at a startup that stored lots of remote sensing data. Their strategy was to put it on a Synology. When the Synology filled up, they bought another, and so forth. Only some of the Synologys were online at any particular time, and there was no indexing to find which Synology held what data. Plus, there were no backups so if one Synology were to blow up, all the data on it wa…

Using Synologys doesn't mean that you have to be stupid about it :-)

Re: Ask HN: How would you store 10PB of data for your startup today?

#124

Earlier quoted context omitted.

The is amazing. Thank you. I’ve been looking at Backblaze storage pods that seem to be designed for that use case. Never rented rack space. Do you remember somehow the math on how much cheaper it was or how you thought about upfront cost vs ongoing. Just order of magnitude would be great.

Roughly a decade ago S3 storage pricing had a ~10x premium over self-hosted. The convenience of not having to touch any hardware is expensive.

Its also important to consider how often disks will fail when you are operating hundreds of them - its probably more often than you'd think, and if you don't have someone on staff and nearby to your colo provider you're going to pay a lot in remote hands fees.

Your colo facility will almost certainly have 24/7 staff on hand who can help you with tasks like swapping disks from a pile of spares, but expect to pay $300+ minimum just to get someone to walk over to your racks, even if the job is 10 mins.

With that said, the cost savings can still be enormous. But know what you're getting into.

Re: Ask HN: How would you store 10PB of data for your startup today?

#125

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

Does Snowball let you exfiltrate data from AWS? I was under the impression it was only for bulk ingestion.

Re: Ask HN: How would you store 10PB of data for your startup today?

#126
post #47

10PB is a crazy amount of data. Far more than any normal business would ever have to deal with. Presuming you aren't crazy, you must have an unusual business plan to legitimately need to handle that much data. That means it's tough for us to say much - any assumptions we might have about it could be invalid depending on your actual business needs. You're just going to have to tell us some more about your business cas…

Unless they do video. In that case 10PB is not much at all.

Re: Ask HN: How would you store 10PB of data for your startup today?

#128

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

Does Snowball let you exfiltrate data from AWS? I was under the impression it was only for bulk ingestion.

First sentence on the linked page: "With AWS Snowball, you pay only for your use of the device and for data transfer out of AWS."

Re: Ask HN: How would you store 10PB of data for your startup today?

#129

Wasabi is a good option. They’re S3 compatible and don’t charge any egress or ingress fees. Been using them for a few years. Great speeds and customer support.

I see from their home page they do not charge for egress, but the FAQ clarifies this is only valid if your monthly egress total is less than or equal to your storage total, otherwise they suspend your service. Should be clarified on the home page in my opinion. At least with an asterisk beside "No egress charges".
Post reply on HN