Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

281–290 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#281

Post by Johnny555 »

Titan2189 wrote: post #259
Johnny555 wrote: post #221

Earlier quoted context omitted.

10PB costs more than $210,000 per month at S3, or more than $12M after five years. Your pricing is off by a 2X - he said he's ok with infrequent access, 1 zone, which is $0.01/GB, or $100K/month. If he rarely needs to read most of the data, he can cut the price by 1/10th by using deep archive, $0.00099 per GB, so $10K/month, or around $600K over 5 years, not including retrieval costs.

Nope, can't use Deep Archive as he specified max retrieval time of 1000ms. But you're correct with S3-IA

For a 10X reduction in cost, things that are impossible often become possible.

Re: Ask HN: How would you store 10PB of data for your startup today?

#282

Post by cortesoft »

closeparen wrote: post #261
walrus01 wrote: post #201

Earlier quoted context omitted.

> The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph, I think at that scale you would want a ceph expert on staff as a full time salaried position. For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it. > EDIT: btw,…

If the colo is far and there’s plenty of headroom, it might not justify much urgency.

That's what remote hands are for. Yes, you batch the replacements up, but this is exactly what remote hands are for.

Re: Ask HN: How would you store 10PB of data for your startup today?

#284

Post by segmondy »

brianwski wrote: post #246
segmondy wrote: post #87

Earlier quoted context omitted.

I think they either stick to S3 or run their own DC with Minio in front. BB as I mentioned in another comment will be a bad idea due to the poor S3 compatible interface, See - https://www.backblaze.com/b2/docs/b2_get_upload_url.html Wasabi might be fine, but don't know if they can handle 10PB.

Disclaimer: I work for Backblaze so I'm biased and you should keep me honest. :-) > Backblaze as I mentioned in another comment will be a bad idea due to the poor S3 compatible interface Backblaze released an S3 compatible API recently: https://www.backblaze.com/b2/docs/s3_compatible_api.html We're ALWAYS curious about any issue customers see, so if there is something specifically missing you use, we both want to hea…

Good to know, how does the presigned URL work? I thought it was this - https://www.backblaze.com/b2/docs/b2_get_upload_url.html. Does it function differently from this? My guess it for a mobile app, the API backend will generate the presigned upload URL hand it off to the mobile app. But certainly don't want the mobile app to have unlimited upload for a 24hr period. So one presigned URL, one upload.

Re: Ask HN: How would you store 10PB of data for your startup today?

#285

Post by craigyk »

ktpsns wrote: post #84

I have worked in HPC (academia) where the cluster storage size is measured in multiples of PB since a decade. Since latency and bandwidth is a killer requirement there, Infiniband (instead of Ethernet) is the defacto standard for connecting the storage pools to the computing nodes. Maintaining such a (storage) cluster requires 1-2 people on site which replace a few hard disks every day. Nevertheless, when I would con…

I am a researcher in academia that handles most of my system admin needs myself. It’s way cheaper to do yourself than some of these comments here make it sound (if you have good server rack space available). I ordered two 60 drive JBODs that I racked by myself (I removed all the drives first to lighten them) for ~82k. I used Zfs and 10 drive raidz2 vdevs for a total capacity of ~960TB of useable file system space. Installing the servers and testing some setups and putting it into use took about 4-5 days. In four years I’ve put many PBs of reAfs and writes through these and had to replace 3 drives. I’d estimate I spend about 2% of my active work focus on maintaining and troubleshooting it. Scaling up to 10PB I’d probably switch to a supported SDS solution, which would be much more expensive, but still way way cheaper than cloud.

Re: Ask HN: How would you store 10PB of data for your startup today?

#286

Post by up2isomorphism »

I have designed, deployed and supported an S3 compatible storage with 5PB capacity for a couple of years, so I acquired the experience to put right hardware and software together together to build such a storage system. And the cost reduction compared to public could like AWS is tremendous. If you are interested in building a private cloud storage for you own, you can contact me at hackernewsantispam@gmail.com for a more detailed discussion.

Re: Ask HN: How would you store 10PB of data for your startup today?

#287

Post by cerved »

cdavid wrote: post #209
skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

[disclaimer: while I have some small experience putting things in DC, including big GPU servers, I have never been anywhere near that scale, certainly not storage] 10k $ is for a server with no hard drive. W/ 12 Gb disks, and with enough RAM, we're talking closer to 40-50k$ per server. Let's say for simplicity you're going to need to buy 15 of those, and let's say you only need to replace 2 of them per year. You need…

yes, the TCO needs consideration, not just the metal

Re: Ask HN: How would you store 10PB of data for your startup today?

#288

Post by cerved »

atomicity wrote: post #273
skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

If you have a PBs of data that you rarely access, it seems to make sense to compress it first. I've rarely seen any non-giants with PBs of data properly compressed. For example, small JSON files converted into larger, compressed parquet files will use 10-100x less space. I am not familiar with images but see no reason why encoding batches of similar images should make it hard to get similar or even better compression…

couple be wrong but I don't believe compression of batches of compressed images compresses well

but it'd be very interested to here about techniques on this because I have a lot of space eaten up by timelapses myself

Re: Ask HN: How would you store 10PB of data for your startup today?

#289

Post by montroser »

user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

> You need to improve your CDN and caching layer.

Depends on usage patterns, but if this is 10PB of users'personal photos and videos, then you're not going to get much value from caching because the hit rate will be so low.

Re: Ask HN: How would you store 10PB of data for your startup today?

#290

Post by dilyevsky »

jedberg wrote: post #216
user5994461 wrote: post #143

Earlier quoted context omitted.

You realize you can't fit 10 appliances of 4U in a rack? (A rack is 42U) There's network equipment and power equipment that requires space in the rack. There's power limitations and weight limitations on the rack that prevents to fill it to the brim.

I've put 39U of drives in a rack before. You only need 1U for a network switch, and you can get power that attaches vertically to the back, so it doesn't take up any space. If you have a cabinet with rack in front and back and all the servers have rails, the weight shouldn't be an issue. The biggest issue will be cooling depending on how hot your servers run. Specifically, it was a rack full of Xserve RAIDs, which ar…

You run a single network switch for a rack full of drives to the brim?
Post reply on HN