RAM?
Lol. A stick of 256gb RAM costs ~$3000. 1TB needs 4. 1PB needs 4000. 10PB needs 40,000. So this would be an upfront cost of $120M.
And this doesn't even cover how you'd fit 40,000 sticks of RAM together.
Ask HN: How would you store 10PB of data for your startup today?
31–40 of 374 posts
Re: Ask HN: How would you store 10PB of data for your startup today?
#32Meta-question: shouldn't there be a website dedicated specifically to reliable, crowd-sourced answers to questions like these? Does it really not exist? I'm thinking like StackShare, but you start from "What's the problem I'm trying to solve?", not "What products are big companies using?".
Re: Ask HN: How would you store 10PB of data for your startup today?
#33Here's an unpopular answer - don't store 10PB of data. Find a way for your startup to work without needlessly having to store insane amounts of data that will likely never be needed.
They're a data storage service.
Re: Ask HN: How would you store 10PB of data for your startup today?
#34Having dealt with a lot of big data I often came to realization that we actually did not need most of it.
Try being intentional and smart in front of your data pipeline and purge data that is not useful. Too many times people store data "just in case" and that case never happens years later.
Re: Ask HN: How would you store 10PB of data for your startup today?
#35900 LTO-U8 tapes
Re: Ask HN: How would you store 10PB of data for your startup today?
#36Earlier quoted context omitted.
Agreed - though I feel like every data use comes after the fact. Original software engineers/developpers rarely have the foresight that the data scientists need the information for (at least imho).
To be fair, the data scientists rarely have the foresight to know what the data scientists need the information for. The only time I've seen a data scientist correctly include all the data they needed (but still be wrong) was when they answered "All of it. We need all of the data".
So true. Tough to know in advance which data will hold the secrets.
Re: Ask HN: How would you store 10PB of data for your startup today?
#37At that level of data you should be negotiating with the 3 largest cloud providers, and going with whoever gives you the best deal. You can negotiate the storage costs and also egress.
Re: Ask HN: How would you store 10PB of data for your startup today?
#38Here's an unpopular answer - don't store 10PB of data. Find a way for your startup to work without needlessly having to store insane amounts of data that will likely never be needed.
Excellent advice for a data backup startup.
Re: Ask HN: How would you store 10PB of data for your startup today?
#39In my opinion, knowing what you're planning to do w/the data once it's stored is the important piece to giving you some idea of where to put it.
Good point. I updated the post with some more infos
Re: Ask HN: How would you store 10PB of data for your startup today?
#40RAM?
Lol. A stick of 256gb RAM costs ~$3000. 1TB needs 4. 1PB needs 4000. 10PB needs 40,000. So this would be an upfront cost of $120M. And this doesn't even cover how you'd fit 40,000 sticks of RAM together.
10PB of RAM being only $120M blows my mind, to be honest. I would have guessed that price was closer to the SSD cost for 10PB.