Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#111

Post by Icer5k »

As others have said, it’s a complicated question, but if you have the resources/wherewithal to run Ceph but don’t want to deal with co-location, you can get a bunch of storage servers from Hetzner and get a much better grasp on cost over S3.

For example, at 10PB with every object duplicated twice (so 20 PB raw storage), you’d need ~90 of their SX293[1] boxes, coming out to around €30k/mo. This doesn’t include time to configure/maintain on your end, but it does cover any costs associated with drive replacement for failure.

I’ve done similar setups for cheap video storage & CDN origin systems before, and it’s worked fairly well if you’re cost conscious.

[1] https://www.hetzner.com/dedicated-rootserver/sx293/configura...

Re: Ask HN: How would you store 10PB of data for your startup today?

#113

Post by staticassertion »

VHRanger wrote: post #102
staticassertion wrote: post #74

Earlier quoted context omitted.

> Update: Should have mentioned earlier, data needs to be accessible at all time. It’s user generated data that is downloaded in the background to a mobile phone, so super low latency is not important, but less than 1000ms required. > The data is all images and videos, and no queries need to be performed on the data. OK, so this definitely helps a bit. At 10PB my assumption is that storage costs are the major thing t…

Isn't Backblaze B2 cheaper than S3?

Yeah, I guess I shouldn't say S3 is the cheapest option there, I was thinking 'In AWS' but Backblaze is cheaper.

Re: Ask HN: How would you store 10PB of data for your startup today?

#114

Post by tw04 »

I should preface this with: I read the question as you want something on-premises/in a colo. If you're talking hosted S3 by someone other than Amazon that's a different story.

It probably depends on if you are tied at the hip to other AWS services. If you are, then you're kind of stuck. The ingress/egress traffic will kill you doing anything with that data anywhere else.

If you aren't, the major players for on-prem S3 (assuming you want to continue access the data that way) would be (in no specific order):

Cloudian

Scality

NetApp Storagegrid

Hitachi Vantara HCP

Dell/EMC ECS

There are plusses and minuses to all of them. At that capacity I would honestly avoid a roll-your-own unless you're on a shoestring budget. Any of the above will be cheaper than Amazon.

Re: Ask HN: How would you store 10PB of data for your startup today?

#116

Post by ilc »

Look at the cost of moving out of the cloud carefully.

Can you afford the up-front costs of the hardware needed to run the solutions you may want to run?

Will those solutions have good enough data locality to be useful to you?

It isn't real useful to have all your data on-site, and then you operations in the cloud. You've introduced many new layers that can fail.

If you go on-prem, the solution to look at is likely Ceph.

Source: Storage Software Engineer, who has spoken at SNIA SDC. I currently maintain a "small" 1PB ceph cluster at work.

Recommendation: Get someone who knows storage and systems engineering to work with you on the project. Even if you decide not to move, understanding why is the most important part.

Re: Ask HN: How would you store 10PB of data for your startup today?

#117

Post by joering2 »

It would be cool to actually have a "blockchain" for something like this. I know the huge amount of data to be store is a niche market, but hear me out:

Everyone that wants to make extra money can join

You join with your computer hooked up to internet, a piece of software running in background

You share % of your hard-drive and limit speed that can be used to upload/download

When someone needs to store 100PB of data ("uploader"), they submit a "contract" on a blockchain - they also setup what's the redundancy rate, meaning how many copies need to be spread to guarantee consistency of data as a whole

The "uploader" shares a file - the file is being chop in chunks and each chunk being encrypted with uploader private PHP key. The info re chunks are uploaded to blockchain and everyone get a piece. In return, all parties that keep piece of uploader data get paid small % either via PayPal or simply in crypto.

I think that would be a cool project, but someone would have to do back-of-napkin number crunching if that would be profitable enough to data hoarders :)

Re: Ask HN: How would you store 10PB of data for your startup today?

#119

Post by x0x0 »

philippb wrote: post #39
garciasn wrote: post #9

In my opinion, knowing what you're planning to do w/the data once it's stored is the important piece to giving you some idea of where to put it.

Good point. I updated the post with some more infos

What is your loss tolerance? If a file is gone, who is annoyed: a free user, a $50/year customer, or a $10k/year customer?

Are these files WORM?

Re: Ask HN: How would you store 10PB of data for your startup today?

#120

Post by throwaway823882 »

TechBro8615 wrote: post #108
philippb wrote: post #65

Earlier quoted context omitted.

Thanks for the pointer. Never thought about this as an option. Great stuff!!!

I’m not an AWS pricing expert, but you should be aware you’re still on the hook for S3 requests even if you can get out of paying for bandwidth. Is AWS direct connect a pure peering arrangement? I wonder what their requirements are for that. Guess I’ll read the link :) Idk what your team’s expertise is, but I’d advise avoiding the cloud as long as possible. If you can build out an on-premise infrastructure, it will b…

I run cloud infra for a living. Have been managing infrastructure for 20 years. I would never for one second consider building my own hosting for a start-up. It would be like a grocery delivery company starting their own farm because seeds are cheap.
Post reply on HN