RAM?
Ask HN: How would you store 10PB of data for your startup today?
11–20 of 374 posts
Re: Ask HN: How would you store 10PB of data for your startup today?
#12Wasabi is a good option. They’re S3 compatible and don’t charge any egress or ingress fees. Been using them for a few years. Great speeds and customer support.
Re: Ask HN: How would you store 10PB of data for your startup today?
#13At this scale, there's no one perfect answer. You need to consider your usage patterns, business needs, etc.
Is the data cold storage, that is rarely accessed? Is it OK to risk losing a percentage of it? Can you identify that percentage? If it's actively utilized, is it all used, or just a subset? Which subset? How much data is added every day? How much is deleted? What are the I/O patterns?
Etc.
I have direct experience moving big cloud datasets to on-site storage (in my case, RAID arrays), but it was a situation where the data had a long-tail usage pattern, and it didn't really matter if some was lost. YMMV.
Re: Ask HN: How would you store 10PB of data for your startup today?
#14In my opinion, knowing what you're planning to do w/the data once it's stored is the important piece to giving you some idea of where to put it.
Agreed - though I feel like every data use comes after the fact. Original software engineers/developpers rarely have the foresight that the data scientists need the information for (at least imho).
Re: Ask HN: How would you store 10PB of data for your startup today?
#15Is the storage of the data critical to the future growth of the business?
[deleted]
Re: Ask HN: How would you store 10PB of data for your startup today?
#16Are you fundamentally a data storage business or are you another business that happens to store a tremendous amount of data?
If it's the former, then investing in-house might make sense (a la Dropbox's reverse course).
Re: Ask HN: How would you store 10PB of data for your startup today?
#17Can you elaborate on what the >10PB of data is and why it’s important to your startup? Is it archived customer data, like backups? Or is it data purchased from vendors for analysis and ML?
Re: Ask HN: How would you store 10PB of data for your startup today?
#18I would host in a datacenter of your choice and do a cross connect into AWS: https://aws.amazon.com/directconnect/pricing/
This allows you to read the data into AWS instances at no cost and process it as needed since there is 0 cost for ingress into AWS. I have some experience with this (hosting using Equinix)
Re: Ask HN: How would you store 10PB of data for your startup today?
#19Wasabi is a good option. They’re S3 compatible and don’t charge any egress or ingress fees. Been using them for a few years. Great speeds and customer support.
10PB with their pricing calculator comes out to over $60,000/mo. Feels like a lot.
edit: perhaps their RCS option would be cheaper if you know exactly how much data you need to store in advance.
Re: Ask HN: How would you store 10PB of data for your startup today?
#20It's going to depend entirely on a number of factors.
How are you storing this data? Is it tons of small objects, or a smaller number of massive objects?
If you can aggregate the small objects into larger ones, can you compress them? Is this 10PB compressed or not? If this is video or photo data, compression won't buy you nearly as much. If you have to access small bits of data, and this data isn't something like Parquet or JSON, S3 won't be a good fit.
Will you access this data for analytics purposes? If so, S3 has querying functionality like Athena and S3 Select. If it's instead for serving small files, S3 may not be a good fit.
Really, at PB scale these questions are all critically important and any one of them completely changes the article. There is no easy "store PB of data" architecture, you're going to need to optimize heavily for your specific use case.