Ask HN: How would you store 10PB of data for your startup today?

Also this is a startup, no? A million or so in storage so you need not preoccupy your startup with having to deal with failing disks, disk provisioning, collocation costs, etc. etc. not to mention the 11 9s of durability you get with S3, to me it just makes the most sense to do this on the cloud.

> downloaded in the background to a mobile phone

and

> but less than 1000ms required

I'm struggling to think of what kind of application needs data access in the background with latency of less than 1000ms. That would normally be for interactive use of some kind.

Getting to 1 min access time would get you into the S3 glacier territory ... you will obviously have considered this but I feel like some really hard scrutiny on requirements could be critical here. With intelligent tiering and smart software you might make a near order of magnitude difference in cost and lose almost no user-perceptible functionality.

1) Staff You'll need at least one, maybe two, to build, operate, and maintain any self-hosted solution. A quick peek on Glassdoor and Salary show the unloaded salary for a Storage Engineer runs $92,000-130,000 US. Multiply by 1.25-1.4 for loaded cost of an employee (things like FICA, insurance, laptop, facilities, etc). Storage Administrators run lower, but still around $70K US unloaded. Point is, you'll be paying around $100K+/year per storage staff position.

2) Facilities (HVAC, electrical, floor loading, etc) If you host on-site (not hosting facility), you'd better make certain your physical facilities can handle it. Can your HVAC handle the cooling, or will you need to upgrade it? What about your electrical? Can you get the increased electrical in your area? How much will your UPS and generator cost? Can the physical structure of the building (floor loading, etc) handle the weight of racks and hundreds of drives, the vibration of mechanical drives, the air cycling?

3) Disaster Recovery/Business Continuity Since you're using S3 One Zone IA, you have no multi-zone duplicated redundancy. It's use case is for secondary backup storage for data, not the primary data store for running a startup. When there is an outage/failure (and it will happen), the startup may be toast, and investors none too happy. So this is another expense you're going to have to seriously consider, whether you stick with S3 or roll-your-own.

4) Cost of money With rolling-your-own, you're going to be doing CAPEX and OPEX. How much upfront and ongoing CAPEX can the startup handle? Would the depreciation on storage assets be helpful financially? You really need to talk to the CPA/finance person before this. There may be better tax and financial benefits by staying on S3 (OPEX). Or not.

Good luck.

The only issue is whether or not you have a CDN in front of this data. If you do then backblaze might not be much cheaper than S3->Cloudfront. You'd save storage costs but easily exceed those savings in egress.

https://www.backblaze.com/blog/open-source-data-storage-serv...

There Storage Pod 6.0 can hold up to 480TB per server.