Off topic, but I'm shocked that anyone would trust uploading sensitive files (e.g. nudes) to this service. Photo vault type apps can be useful, but I would never want the content in those apps to upload to a small service like this based on their word that employees won't go through it.
Ask HN: How would you store 10PB of data for your startup today?
61–70 of 374 posts
Re: Ask HN: How would you store 10PB of data for your startup today?
#62Re: Ask HN: How would you store 10PB of data for your startup today?
#63Earlier quoted context omitted.
No. They are already managing 10PB, planning for which would be very stupid when just starting up.
Why would planning to be able to execute the single focus of your startup be stupid?
I tend to agree. If you are a storage company, I’d think that part of your secret sauce should be how to store tens or hundreds of petabytes of customer backups economically.
Maybe I’m wrong though. Perhaps the real secret sauce is the end user experience and the kind of storage you use on the backend doesn’t matter at all.
However I bet that the “cloud storage space” is pretty crowded and lots of people shop on price more than anything. If your business model is all about price, then finding economical storage is critical to your company and needs to be part of your core competency.
If price isn’t that important, perhaps it doesn’t matter... the “winners” would win no matter how expensive their storage solution is.
But honestly.... I feel like part of your core competency needs to be managing the storage system.
Re: Ask HN: How would you store 10PB of data for your startup today?
#64Earlier quoted context omitted.
To be fair, purchasing and hosting even the most basic mirrored RAID array of that scale comes to well over half a million for the disks alone. Then you need to manage them.
10x Supermicro SSG-6049P-E1CR60H servers (60 x 3,5" HDD in 4U enclosure) - $5k each 600x WESTERN DIGITAL Ultrastar DC HC550 18TB (10800PB in total) - $500 each ~$350k in hardware, up to 20kW energy consumption, should fit in two rack towers. You can host it for about $1.5k somewhere. All assuming no redundancy :)
Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning.
So toss in at least one SRE type person. Say $200k/year.
Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one....
Since redundancy is a thing, double that $350k. And 10pb is what they have now so double it again for 20pb. Add in $10k per rack for switches, routers, wires, etc.
So probably you are looking at a million dollars of capital plus labor to actually execute on this. And don’t forget the lead time might be a month to get the hardware and a week or two to install it. Plus all the configuration management that needs to be built up. Not to mention monitoring. So maybe a quarter of work just to have it functional.
I haven’t even factored in opportunity costs. What could this business be doing that adds more value than building out a little data center?
I dunno. Maybe it does make sense to manage your own hardware. But it helps to calculate the entire cost of ownership, not just the cost of the servers.
Re: Ask HN: How would you store 10PB of data for your startup today?
#65I would host in a datacenter of your choice and do a cross connect into AWS: https://aws.amazon.com/directconnect/pricing/ This allows you to read the data into AWS instances at no cost and process it as needed since there is 0 cost for ingress into AWS. I have some experience with this (hosting using Equinix)
Thanks for the pointer. Never thought about this as an option. Great stuff!!!
Re: Ask HN: How would you store 10PB of data for your startup today?
#66If you have good sysadmin/devops types, this is a few racks of storage in a datacenter. Ceph is pretty good at managing something this size, and offers an S3 interface to the data (with a few quirks). We were mostly storing massive keys that were many gigabytes, so if you have smaller keys, so I'm not sure about performance/scalding limits with smaller keys and 10PB. I'd be sure to give your team a few months to buil…
The is amazing. Thank you.
I’ve been looking at Backblaze storage pods that seem to be designed for that use case.
Never rented rack space.
Do you remember somehow the math on how much cheaper it was or how you thought about upfront cost vs ongoing. Just order of magnitude would be great.
Re: Ask HN: How would you store 10PB of data for your startup today?
#67It seems to me like you could save a ton of money by using your own hardware. Perhaps buy a bunch of big Synology boxes? At that scale you should also consider looking at technologies such as Ceph. We've recently switched to a setup with several Synology boxes for around 1PB net storage.
Thank you!
Re: Ask HN: How would you store 10PB of data for your startup today?
#68Can you elaborate on what the >10PB of data is and why it’s important to your startup? Is it archived customer data, like backups? Or is it data purchased from vendors for analysis and ML?
See updated question. Thanks for asking
Re: Ask HN: How would you store 10PB of data for your startup today?
#69Earlier quoted context omitted.
10x Supermicro SSG-6049P-E1CR60H servers (60 x 3,5" HDD in 4U enclosure) - $5k each 600x WESTERN DIGITAL Ultrastar DC HC550 18TB (10800PB in total) - $500 each ~$350k in hardware, up to 20kW energy consumption, should fit in two rack towers. You can host it for about $1.5k somewhere. All assuming no redundancy :)
Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning. So toss in at least one SRE type person. Say $200k/year. Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... Since redundancy is a thing, double that $350k. And 10pb is what they…
Thanks for laying this out. Never rented rack space my self
Re: Ask HN: How would you store 10PB of data for your startup today?
#70Earlier quoted context omitted.
If you can solve that for me without affecting revenue I have $1m in cash for you right there. We are a photo/video storage service.
So from a completely evil (well, capitalist) perspective, do you have data on how often people retrieve backups, and at what 'age' they do so? Because there may be an inflection point that offering monetary compensation for data loss, rather than actually trying to store the data, would make more financial sense. I.e., "All data > than 2 years gets silently expunged, and anyone trying to retrieve it at that point get…
(A less evil approach that might still lead to reduced costs would be detecting old, unaccessed data of sufficient size and flagging it for users, with a small refund or service discount if they purge it. Though that assumes you have 'power users' who are storing massive amounts, to where the savings in storage costs would be worth it)
(And if you don't already, I would also consider making it so items that are in the trash for some period of time, say 30 days, get deleted automatically as well, possibly with a reminder email a few days before)
(And lastly, depending on user profiles and usage, incentives around reducing resolution/quality of photos and video, and automating that in the app as part of the sync process, might provide some opportunities to reduce costs of storage > the lost revenue of cheaper plans.)