Ask HN: How would you store 10PB of data for your startup today?

I have no idea how you evaluate the necessity of keeping the data safe, and that plays a huge factor in deciding what's appropriate. Amazon S3 makes it a no-brainer for having your data safe across failure domains. Of course, the same can be done with non-S3 solutions, but someone has to set it all up, test it, and pay for it.

My background in storage is mostly related to working with Ceph and Swift (both OpenStack Swift and SwiftStack) while being employed by various hardware vendors.

Some thoughts on Ceph: - In my opinion, Ceph is better suited for block storage than object storage. To be fair, it does support object storage with use of the Rados Gateway (RGW) and RGW does support the S3 API. However, Ceph has a strong consistency model and in my opinion, strong consistency tends to be better suited to block storage. Why is this? For a 10PB cluster (or larger), failures of various types will be the norm (mostly disk failures). What does Ceph do when a disk fails? It goes to work right away to move whatever data was on the failed disk (using its redundant copies/fragments) to a new place. No big deal if it's only a single HDD that's in failed status at any given point of time. What if you have a server, disk controller, or drive shelf fail? You get a whole bunch of data backfilling going on all at once. The other consideration with strong consistency model is having multi-site storage. Not so good for strong consistency model (due to higher latency for inter-site communication). - Ceph has a ton of knobs, is very feature rich, and high on complexity (although it has improved). The open-source mechanisms for installing and the admin tools have experienced (and continue to have) a high-rate of churn. Do a quick search on how to install/deploy Ceph and you'll see multiple. Same with admin tools. Should you strongly consider Ceph as an option, I would strongly advise you to license and use one of the 3rd party software suites that (a) take the pain away from install/deploy/admin, and (b) reduce the amount of deep expertise that you would need to keep it running successfully. Examples of these 3rd party Ceph admin suites are Croit [0] and OSNEXUS [1]. Alternatively, if you like the idea of a Ceph appliance, I would take a close look at SoftIron [2].

Aside from Ceph, it's worth taking a very close look at OpenStack Swift [3][4]. It's only object storage and has been around for about 10 years. It supports the S3 protocol and also has its own Swift protocol. It's open source and it has an eventually consistent data model. Eventually consistent is (IMO) a much better fit for a 10+PB cluster of objects. Why is this? Because failures can be handled with less urgency and at more opportune times. Additionally, an eventually consistent model makes multi-site storage MUCH easier to deal with.

I suggest going further and spending some quality time with the folks at SwiftStack [5]. Object storage is their game and they're very good at it. They can also help with on-prem vs hosted vs hybrid deployments.

Additionally, you would definitely want to use erasure coding (EC) as opposed to full replication. This is easy enough to do with either Swift or Ceph.

Disclaimers and disclosures - I am not currently (nor have ever been) employed by any of the companies I mentioned above.

Dell EMC Technical Lead and co-author of these documents:

   Dell EMC Ready Architecture for Red Hat Ceph Storage 3.2 - Object Storage Architecture [6]
   Dell EMC Ready Architecture for SwiftStack Storage - Object Storage Architecture Guide [7]

   "Accelerating Swift with Intel Cache Acceleration Software" [8]


   [0] https://croit.io
   [1] https://www.osnexus.com/technology/ceph
   [2] https://softiron.com
   [3] https://wiki.openstack.org/wiki/Swift
   [4] https://github.com/openstack/swift
   [5] https://www.swiftstack.com
   [6] https://www.delltechnologies.com/resources/en-us/asset/technical-guides-support-information/solutions/red_hat_ceph_storage_v3-2_object_storage_architecture_guide.pdf
   [7] https://infohub.delltechnologies.com/section-assets/solution-brief-swiftstack-1
   [8] https://www.intel.sg/content/www/xa/en/software/intel-cache-acceleration-software-performance/intel-cache-acceleration-software-performance-accelerating-swift-white-paper.html

And it already supports S3 API, and other HTTP, FUSE, WebDAV, Hadoop, etc.

There should be many existing hardware options that is much cheaper than AWS S3.

Or a 648T raw HDD storage box for ~$53k

To get that up to raw 10 PB, I need ~$2m for all-SSD, or ~$850k for all-HDD

Bake-in a 2-system safety margin, and that's ~$2.3m all-SSD or ~$960 all-HDD

Run TrueNAS and ZFS on each of them ... and my overhead becomes a little bit of cross-over sysadmin/storage admin time per year and power

Say that's 1 FTE at $180k ($120k salary + 50% overhead) per year (even though actual admin time is only going to be maybe 10% of their workload - I like rounding-up for these types of approximations)

Peak cost, therefore, is ~$2.5m the first year, and ~$200k per year afterwards

And, of course, we'll want to plan for replacement systems to pop-in ... so factor-up to $250k per year in overhead (salary, benefits, taxes, power, budget for additional/replacement servers)

Using [Wasabi](https://wasabi.com/cloud-storage-pricing/#three-info), 10PB is going to run ~$62k/mo, or ~$744k per year

It's cheaper to build-vs-buy in no more than 5 years ... probably under 3