Check whether you really need 10 PB or you can make do with several orders of magnitude less. I wouldn't be surprised if it was some sort of perverse incentive CV building thing, like engineers building a Kubernetes cluster for every tiny thing. If you really do need 10 PB, then still you probably should check again because you probably don't need 10 PB.
Ask HN: How would you store 10PB of data for your startup today?
151–160 of 374 posts
Re: Ask HN: How would you store 10PB of data for your startup today?
#152If you have good sysadmin/devops types, this is a few racks of storage in a datacenter. Ceph is pretty good at managing something this size, and offers an S3 interface to the data (with a few quirks). We were mostly storing massive keys that were many gigabytes, so if you have smaller keys, so I'm not sure about performance/scalding limits with smaller keys and 10PB. I'd be sure to give your team a few months to buil…
Totally out-of-band for this thread, but... what are the uses for a multi-gigabyte key?! I'm clearly unaware of some cool tech, any key words I can search?
I'm no expert but I would guess it's just a fancy word for "file", as in "key-value store", as opposed to a god-proof encryption key.
Re: Ask HN: How would you store 10PB of data for your startup today?
#153Cloud or self-hosted will depend on your in-house expertise. For cloud others have already mentioned Backblaze and Wasabi, but you can also check Scaleway, they do 0.02 EUR/GB/mo for hot storage and 0.002/GB/mo for cold storage.
Since we're talking about images and videos, do you already have different quality of each media available? Maybe thumbnail, high quality, and full quality. It could allow you to use cold storage for the full quality media, serving the high quality version while waiting for retrieval.
If the use case is more of a backup/restore service and a restore typically takes longer than a cold storage retrieval (being Glacier or self hosted tape robot), then keep just enough in S3 to restore while you wait for the retrieval of the rest.
If you go the self-hosted route, I like software that is flexible around hardware failures. Something that will rebalance automatically and reduce the total capacity of the cluster, rather than require you to swap the drive ASAP. That way you can keep batch all the hardware swapping/RMA once per week/month/quarter.
Re: Ask HN: How would you store 10PB of data for your startup today?
#154Earlier quoted context omitted.
I’m not an AWS pricing expert, but you should be aware you’re still on the hook for S3 requests even if you can get out of paying for bandwidth. Is AWS direct connect a pure peering arrangement? I wonder what their requirements are for that. Guess I’ll read the link :) Idk what your team’s expertise is, but I’d advise avoiding the cloud as long as possible. If you can build out an on-premise infrastructure, it will b…
I run cloud infra for a living. Have been managing infrastructure for 20 years. I would never for one second consider building my own hosting for a start-up. It would be like a grocery delivery company starting their own farm because seeds are cheap.
Depends what you’re doing I suppose. I think the three companies I mentioned (CloudFlare, Zoom and WhatsApp) are good examples of infrastructure investment as a competitive advantage.
Re: Ask HN: How would you store 10PB of data for your startup today?
#155Earlier quoted context omitted.
No. They are already managing 10PB, planning for which would be very stupid when just starting up.
Why would planning to be able to execute the single focus of your startup be stupid?
I also tend to agree. I think AWS is great and use it as my default solution, but if I was starting a company that had high bandwidth and/or storage requirements, I would be looking for other solutions from day one.
Re: Ask HN: How would you store 10PB of data for your startup today?
#156Earlier quoted context omitted.
Wow that’s up to $500,000 just to export 10PB (depending on region).
According to https://aws.amazon.com/snowball/pricing/ , egress fees depends on the region, which can range from $0.03/GB (North America & parts of Europe) to $0.05/GB (parts of Asia and Africa). So US$300K to US$500K for egress fees + cost of Snowball devices. The major downside of Snowball in this export case is the size limit of 80TB per device - from https://aws.amazon.com/snowball/features/ : "Snowball Edge Stora…
There's also the snowmobile https://aws.amazon.com/snowmobile/
Re: Ask HN: How would you store 10PB of data for your startup today?
#157Earlier quoted context omitted.
Wow that’s up to $500,000 just to export 10PB (depending on region).
According to https://aws.amazon.com/snowball/pricing/ , egress fees depends on the region, which can range from $0.03/GB (North America & parts of Europe) to $0.05/GB (parts of Asia and Africa). So US$300K to US$500K for egress fees + cost of Snowball devices. The major downside of Snowball in this export case is the size limit of 80TB per device - from https://aws.amazon.com/snowball/features/ : "Snowball Edge Stora…
[deleted]
Re: Ask HN: How would you store 10PB of data for your startup today?
#158Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed servic…
You can also get units like this direct from Western Digital/HGST. We have a system with 3 of their 4U60 units, and they weren't all that expensive. Ordering direct from HGST, we only paid a small premium on top of the cost of the SAS drives.
Re: Ask HN: How would you store 10PB of data for your startup today?
#159At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…
The thing about fitting everything in one rack, potentially, is vibration. There have been several studies into drive performance degredation from vibration, and there's noticeable impact in some scenarios. The Open Compute "Knox" design as used by Facebook spins drives up when needed, and then back down, though whether that's for vibration impact, I don't know (their cold storage use [0]). 0: https://datacenterfront…
Here is Brendan Gregg showing how vibrations can affect disk latency:
Re: Ask HN: How would you store 10PB of data for your startup today?
#160Earlier quoted context omitted.
Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning. So toss in at least one SRE type person. Say $200k/year. Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... Since redundancy is a thing, double that $350k. And 10pb is what they…
> Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... This person's entire job is managing a few racks of hard drives? How often do you think they're actually going to get called in? > Since redundancy is a thing, double that $350k. True, but you can do redundancy for cheaper with parity or tape. > And 10pb is what they have no…
> True, but you can do redundancy for cheaper with parity or tape.
At this volumes you probably do want a carbon copy at another site to mitigate disasters like datacenter fires.