Ask HN: How would you store 10PB of data for your startup today? • Hacker News Board

Re: Ask HN: How would you store 10PB of data for your startup today?

#371

Post by warrenm » Wed, May 05, 2021, 8:18 PM UTC

skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

>RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. (https://www.rackmountpro.com/product.php?pid=3154)

I dare you to buy 102 12TB drives for $11k

The cheapest consumer class 12GB hdd is ~$275 a pop

That's $28k just for the drives

Re: Ask HN: How would you store 10PB of data for your startup today?

#372

Post by warrenm » Wed, May 05, 2021, 8:20 PM UTC

fakedang wrote: post #235

pokler wrote: post #159

Earlier quoted context omitted.

Here is Brendan Gregg showing how vibrations can affect disk latency: https://www.youtube.com/watch?v=tDacjrSCeq4

I'm an absolute noob here, but is using SSD racks for storage a feasible option cost wise and for this issue in particular?

Absolutely it's an option

It's gonna cost more

But it's also going to be nearly vibration-free (just the PSU fans), and stupidly-fast

Re: Ask HN: How would you store 10PB of data for your startup today?

#373

Post by warrenm » Wed, May 05, 2021, 8:34 PM UTC

creiht wrote: post #283

Late to the party, but one does not simply store 10PB of data :) The short story is, ignore most of the advice, poach^H^H^H^H^Hhire someone who has done this, and leverage their expertise. There is no armchair quarterbacking infrastructure at this scale.

Honestly, 10PB is small potatoes nowadays - and it's not "armchair quarterbacking" to think about storing it in a distributed environment

I work with one customer right now that's storing something like 27PB across ~100 systems for analysis on a rolling 90-to-365-day period (and they're a relatively small customer) with Splunk

If they were doing more storage than analysis, that storage cluster would be substantially smaller

Re: Ask HN: How would you store 10PB of data for your startup today?

#374

Post by warrenm » Wed, May 05, 2021, 8:38 PM UTC

notyourday wrote: post #126

ufmace wrote: post #47

10PB is a crazy amount of data. Far more than any normal business would ever have to deal with. Presuming you aren't crazy, you must have an unusual business plan to legitimately need to handle that much data. That means it's tough for us to say much - any assumptions we might have about it could be invalid depending on your actual business needs. You're just going to have to tell us some more about your business cas…

Unless they do video. In that case 10PB is not much at all.

It's not merely video

Machine data takes a lot of space, depending on how long you need/want to hold onto it - any given even might only be a few 100 bytes, but with 1000s or 10s of 1000s of devices sending even syslog turns into a metric buttload of data :)