Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

251–260 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#251

Post by shiftpgdn »

user5994461 wrote: post #143
skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

You realize you can't fit 10 appliances of 4U in a rack? (A rack is 42U) There's network equipment and power equipment that requires space in the rack. There's power limitations and weight limitations on the rack that prevents to fill it to the brim.

Gold standard APC PDUs are all 0U side mount.

Re: Ask HN: How would you store 10PB of data for your startup today?

#252

Post by shiftpgdn »

ktpsns wrote: post #84

I have worked in HPC (academia) where the cluster storage size is measured in multiples of PB since a decade. Since latency and bandwidth is a killer requirement there, Infiniband (instead of Ethernet) is the defacto standard for connecting the storage pools to the computing nodes. Maintaining such a (storage) cluster requires 1-2 people on site which replace a few hard disks every day. Nevertheless, when I would con…

I've done multiple multipetabyte scale projects and you only need to swap disks once a month or so. I had a project (as a solo engineer) 2 hours away and I drove there once in six months.

Re: Ask HN: How would you store 10PB of data for your startup today?

#253

Post by gamedna »

Context please.

1. Do you have paying customers already?

2. Can the startup weather large capex? does opex work better for you?

3. Do you already have staff with sufficient bandwidth to support this, or will you need to hire?

4. What are the access patterns for the data?

5. What is the data growth rate?

6. What is the cost of losing some, or all of this data?

7. What is your expected ROI?

TL;DR - storing and serving up the data is the easy part.

Re: Ask HN: How would you store 10PB of data for your startup today?

#254

Post by shiftpgdn »

sethhochberg wrote: post #124
mceachen wrote: post #88

Earlier quoted context omitted.

Roughly a decade ago S3 storage pricing had a ~10x premium over self-hosted. The convenience of not having to touch any hardware is expensive.

Its also important to consider how often disks will fail when you are operating hundreds of them - its probably more often than you'd think, and if you don't have someone on staff and nearby to your colo provider you're going to pay a lot in remote hands fees. Your colo facility will almost certainly have 24/7 staff on hand who can help you with tasks like swapping disks from a pile of spares, but expect to pay $300+…

I currently pay about $40 for a half hour of remote hands at a large data center. Modern disks rarely need to be swapped. You can look at BackBlaze's published failure rates and do the math yourself if you don't believe me.

Re: Ask HN: How would you store 10PB of data for your startup today?

#256

Post by shiftpgdn »

oneplane wrote: post #222
marcinzm wrote: post #184

Earlier quoted context omitted.

Like another comment said, don't bother swapping out disks, just leave the dead ones in place and disable them in software. Then eventually either replace the whole server or get someone on site to do a mass swap of disks. At this scale redundancy needs to be spread between machines anyway so no gain in replacing disks as they die.

That also means that you need extra spare disks in the system, which also means extra servers, extra racks, extra power feeds, extra cooling etc. If you do a 60-disk 4U setup you'll need 1 full rack of those just to get your 10PB, then you'll need yet another one for redundancy. And then a quarter for hot spares. At that point you have single-redundancy, no file history and no scaling. Is it possible? Sure. Is this s…

You don't do redundancy that way at that scale, that's completely insane. You run ceph or beegfs or Windows Storage Server and backup to tape with a tape library. If youve got big bucks (though still peanuts compared to s3) you replicate the entire setup 1:1 at a second site.

Re: Ask HN: How would you store 10PB of data for your startup today?

#257

Post by dsyrk »

philippb wrote: post #65
jtchang wrote: post #18

I would host in a datacenter of your choice and do a cross connect into AWS: https://aws.amazon.com/directconnect/pricing/ This allows you to read the data into AWS instances at no cost and process it as needed since there is 0 cost for ingress into AWS. I have some experience with this (hosting using Equinix)

Thanks for the pointer. Never thought about this as an option. Great stuff!!!

I’d like to add I’d agree with the parent comment and add some specifics.

Buy storage servers from 45drives they basically build same hardware as Backblaze uses. Add copper 10G nics to the servers.

https://www.45drives.com/

Get necessary switches 10G with 40G uplink ports. Whatever your favorite. Use 10GBaseT to the servers.

Install hardware in a quality data center. Like one of theirs -

https://www.digitalrealty.com/

And get 10G virtual cross connects to AWS.

Back of the envelope calculation you need 30TB raw, so about 60 servers. They aren’t really that power hungry so 10 per cabinet. 6 cabinets. at least 6+2 switches.

Software wise you have lots of options with this infra. High upfront cost but low MRC vs all other options. Assuming you have skilled sys admins who know what they are doing.

Re: Ask HN: How would you store 10PB of data for your startup today?

#258

Post by mattgair »

erulabs wrote: post #197

I’d go with Ceph and dedicated hardware. Something like Hetzner or Datapacket, or built it yourself and go big with something like SoftIron. We’ve built and maintain a number of these types of clusters - using S3 compatible APIs (CephObjectStore). SoftIron is probably overkill but good lord is it fun to play with that much thruput! If you’re looking for a partner/consultant to get things going, feel free to reach out…

SoftIron would love to assist with this.

Re: Ask HN: How would you store 10PB of data for your startup today?

#259

Post by Titan2189 »

Johnny555 wrote: post #221
skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

10PB costs more than $210,000 per month at S3, or more than $12M after five years. Your pricing is off by a 2X - he said he's ok with infrequent access, 1 zone, which is $0.01/GB, or $100K/month. If he rarely needs to read most of the data, he can cut the price by 1/10th by using deep archive, $0.00099 per GB, so $10K/month, or around $600K over 5 years, not including retrieval costs.

Nope, can't use Deep Archive as he specified max retrieval time of 1000ms. But you're correct with S3-IA

Re: Ask HN: How would you store 10PB of data for your startup today?

#260

Post by quantumofalpha »

user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

AWS is ridiculously expensively at their scale, both for storage and egress. But the choice is not only between that and building a staffed on-premise storage facility.

You can compromise at a middle ground - rent a bunch of VPS/managed servers and let the hosting companies deal with all the nastiness of managing physical hardware and CAPEX. Cost around $1.6-2/TB/month (e.g. Hetzner's SX) for raw non-redundant storage, an order of magnitude better than AWS. Comes with far more reasonably priced bandwidth too.

Build some error correction on top using one of the many open-source distributed filesystems out there or perhaps an in-house software solution (reed-solomon isn't exactly rocket science). And for some 30+% overhead, depending on workload (you can have very low overload if you have few reads or very relaxed latency requirements), you should have a decently fault tolerant distributed storage at a fraction of AWS costs.

Post reply on HN