Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

211–220 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#211

Post by shyn3 »

pier25 wrote: post #204
user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

> If it's the bandwidth. You need to improve your CDN and caching layer. What would you recommend for this? (considering data is stored in S3)

Verizon and Redis has worked well for me.

Re: Ask HN: How would you store 10PB of data for your startup today?

#212

Post by msk20 »

I don't really know much about optimizing storage costs, But You could learn from storage giants.

Example is Blackblaze storage pod 6.0 according to them it holds 0.5PB with a cost of 10k$, you will need about 20*10K$ = 200K$ + Maintenance(They also publish failure rates) , The schematics and everything is in their website and according to them they have already a supplier who provides them with such devices which you could probably buy from. Note: This was published 2016, they probably have Pod 7.0 by now so cost may be better.

Reference: https://www.backblaze.com/blog/open-source-data-storage-serv...

Re: Ask HN: How would you store 10PB of data for your startup today?

#215

Post by tempest_ »

pas wrote: post #206
dividedbyzero wrote: post #176

Earlier quoted context omitted.

Does it scale that far?

Likely not that "gracefully". But Ceph absolutely does, and has an S3 gateway.

Using Ceph like S3 could be a bit tricky if all of that 10pb is very small files.

Redhat did an interesting series of blog posts about Ceph and getting it to 1 billion objects

https://www.redhat.com/en/blog/scaling-ceph-billion-objects-...

Re: Ask HN: How would you store 10PB of data for your startup today?

#216

Post by jedberg »

user5994461 wrote: post #143
skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

You realize you can't fit 10 appliances of 4U in a rack? (A rack is 42U) There's network equipment and power equipment that requires space in the rack. There's power limitations and weight limitations on the rack that prevents to fill it to the brim.

I've put 39U of drives in a rack before. You only need 1U for a network switch, and you can get power that attaches vertically to the back, so it doesn't take up any space. If you have a cabinet with rack in front and back and all the servers have rails, the weight shouldn't be an issue.

The biggest issue will be cooling depending on how hot your servers run.

Specifically, it was a rack full of Xserve RAIDs, which are 3U each and about 100lbs each. So that was over 1300lbs.

Re: Ask HN: How would you store 10PB of data for your startup today?

#217

Post by artemist »

You almost certainly should not have 10PB of data. Not just is it extremely expensive, it is unlikely that millions of people have each allowed you to take gigabytes of their data. You are sitting on a huge violation of CCPA, GDPR, and other privacy laws, as well as copyright issues. If you are scraping data off the Internet you likely have content illegal to poses in several different countries (such as child sexual abuse material or videos of ISIL killings). As a startup you do not have the legal and technical capabilities to manage this data so you should not have it.

Re: Ask HN: How would you store 10PB of data for your startup today?

#218

Post by chubot »

Why not downsample everything to 10% the size, put those online, and use Amazon Glacier for the originals? (e.g. for exporting)

If you're storing images and videos directly from the phone, they can be downsampled drastically without losing quality on a viewing device that anyone's likely to have.

It's unlikely that anyone wants to download the full size copy, and if they do, they can wait a few hours for Glacier.

You could expose this to the customer, e.g. offer direct access of originals at 2x or 5x the price. But 99.9% of people will be OK with immediate access to quality images/video and eventual access to the unmodified originals.

Re: Ask HN: How would you store 10PB of data for your startup today?

#219

Post by brunick »

artemist wrote: post #217

You almost certainly should not have 10PB of data. Not just is it extremely expensive, it is unlikely that millions of people have each allowed you to take gigabytes of their data. You are sitting on a huge violation of CCPA, GDPR, and other privacy laws, as well as copyright issues. If you are scraping data off the Internet you likely have content illegal to poses in several different countries (such as child sexual…

a short research shows, this is the cofounder of keepsafe, so i guess they most likely got the data from their customers

Re: Ask HN: How would you store 10PB of data for your startup today?

#220

Post by pmorici »

I'd look at using a Storinator cluster with a scalable network filesystem like Gluster, Lustre, Ceph or something along those lines. A 4U Storinator with 60 18TB drives has 1PB of raw capacity and cost $43,0000. You'd been looking at a upfront cost of $500k but if you amortize that cost over a 5 year period you are looking at 100k per year plus you are going to need someone that dedicates an amount of time to maintaining that.
Post reply on HN