Ask HN: How would you store 10PB of data for your startup today?

The right answer for you may have more to do with your business requirements than technical requirements. I've done large scale storage in cloud providers (S3, GCS, etc.) and on premise (I designed the early storage systems at Dropbox). I haven't found there to be a one-size-fits-all answer.

If you place a high value on engineering velocity and you already rely on managed services, then I would look to stay in S3. Do the legwork to gather competitive bids (GCS, Azure, maybe one second tier option) and use that in your price negotiation. Negotiation is a skill, so depending on the experience in your team, you may have better or worse results -- but it should be possible to get some traction if you engage in good faith with AWS.

There is a considerable opportunity cost to moving that data to another cloud provider. No matter how well you plan and execute it, you're going to lose some amount of velocity for at least several months. In a worse scenario, you are running two parallel systems for a considerable amount of time and have to pay that overhead cost on your engineering team's productivity. In the worst case scenario, you experience service degradation or even lose customer data. It's quite easy for 2-3 months to turn into 2-3 years when other higher priority requirements appear, and it's also easy for unknowns to pop up and complicate your migration.

With all of that said, if the fully baked cost of migrating to another cloud provider (engineering time + temporary migration services + a period of duplicated costs between services + opportunity cost) is trajectory changing for your business, then it certainly can be done. I feel like GCS is a bit better of a product vs S3, although S3 has managed to iron out some of its legacy cruft in the last few years. Azure is not my cup of tea. I have never seriously considered any other vendors in the space, although there are many.

Your other option is to build it. I've done it several times, people do it every day. You may need someone on the team who either has or can grow the skillset you're going to need: vendor negotiation, capacity planning, hardware qualification, and other operational tasks. You can save a bunch of money, but the opportunity cost can be even greater.

10PB is the equivalent of maybe 1-2 rack of servers in a world where you can easily get 40-50 drive systems with 10-18TB drives (of course for redundancy you would need more like 2-2.5x, and you need space to grow into so that you're always ahead of your user growth curve). At any rate, my point is that the deployment isn't particularly large, so you aren't going to see good economies of scale. If you expect to be in the 100+PB range in 6-12 months, this could still be the right option.

Personally, I would look to build a service like this in S3 and migrate to on-premise at an inflection point probably 2 orders of magnitude above yours, if the future growth curve dictated it. The migration time and cost will be even more onerous, but the flexibility while finding product/market fit probably countermands the cost overhead.

There is a third option, which is hosted storage where someone else runs the machines for you. Personally I see it as a stop-gap solution on the path to running the machines yourself, and so it's not very exciting. But it is a way to minimize your investment before fully committing.

Use Intelligent tiering or some kind of a custom system that moves data into glacier more aggressively based on access times.It can help a lot.

Disclaimer: I work for Backblaze so I'm biased and you should keep me honest. :-)

> Backblaze as I mentioned in another comment will be a bad idea due to the poor S3 compatible interface

Backblaze released an S3 compatible API recently: https://www.backblaze.com/b2/docs/s3_compatible_api.html

We're ALWAYS curious about any issue customers see, so if there is something specifically missing you use, we both want to hear about it, and we might be able to add it. Even if it doesn't help you right away, maybe it will help somebody else a few months down the road who might need the same feature.

We know we're compatible with Veeam backups (which only go through S3 APIs) for instance, and we continue to maintain that. We added the "S3 Object Lock" specifically for this particular vendor. So if you are missing one or two APIs, let us know!

Been there, done that, at all levels. I would much rather be working on a 10PB set of hardware racks, including all the drive replacements. When you factor in the costs of compute hardware (to make that useful), networking equipment, etc, it's trebled again the cost, and then again for the power, cooling, and cage space to run it all. The actual break-even point of running your own hardware is more like 2 years.

But it's not about price: It's about control, and it's about the expertise you gain from running all of that. If you have 10PB of data, you should have someone in-house who knows how to work with 10PB of data at a low level, and the best way to get that is to employ people at all levels to make that work. You gain significant advantage from having the direct performance data and the expertise of having techs whose 9-5 is replacing disks.

Disclaimer: I work for Backblaze so I'm biased.

> Have you tried backblaze b2 storage? Requires more work client-side...

Backblaze recently released an S3 compatible API, so I'm hoping it is zero client-side work: https://www.backblaze.com/b2/docs/s3_compatible_api.html

If you try it, and find any issues, please let us know!