Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

221–230 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#221

Post by Johnny555 »

skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

10PB costs more than $210,000 per month at S3, or more than $12M after five years.

Your pricing is off by a 2X - he said he's ok with infrequent access, 1 zone, which is $0.01/GB, or $100K/month.

If he rarely needs to read most of the data, he can cut the price by 1/10th by using deep archive, $0.00099 per GB, so $10K/month, or around $600K over 5 years, not including retrieval costs.

Re: Ask HN: How would you store 10PB of data for your startup today?

#222

Post by oneplane »

marcinzm wrote: post #184
sethhochberg wrote: post #124

Earlier quoted context omitted.

Its also important to consider how often disks will fail when you are operating hundreds of them - its probably more often than you'd think, and if you don't have someone on staff and nearby to your colo provider you're going to pay a lot in remote hands fees. Your colo facility will almost certainly have 24/7 staff on hand who can help you with tasks like swapping disks from a pile of spares, but expect to pay $300+…

Like another comment said, don't bother swapping out disks, just leave the dead ones in place and disable them in software. Then eventually either replace the whole server or get someone on site to do a mass swap of disks. At this scale redundancy needs to be spread between machines anyway so no gain in replacing disks as they die.

That also means that you need extra spare disks in the system, which also means extra servers, extra racks, extra power feeds, extra cooling etc.

If you do a 60-disk 4U setup you'll need 1 full rack of those just to get your 10PB, then you'll need yet another one for redundancy. And then a quarter for hot spares. At that point you have single-redundancy, no file history and no scaling. Is it possible? Sure. Is this something you can do 'on a side track with the people you already heave'? Unlikely if you are a startup with no datacenter, no colocation yet etc.

Re: Ask HN: How would you store 10PB of data for your startup today?

#223

Post by alfalfasprout »

glbrew wrote: post #121
ktpsns wrote: post #84

I have worked in HPC (academia) where the cluster storage size is measured in multiples of PB since a decade. Since latency and bandwidth is a killer requirement there, Infiniband (instead of Ethernet) is the defacto standard for connecting the storage pools to the computing nodes. Maintaining such a (storage) cluster requires 1-2 people on site which replace a few hard disks every day. Nevertheless, when I would con…

Since he needs 1000ms response on storage isn't ethernet the better option? It can reach 400gb/s on fastest hardware now. I thought Infiniband was only reasonable to use when machines need to quickly access other machines primary memory. I would like to know if I'm wrong about this though.

Agreed and at this point with ROCE there's little reason to go with infiniband given you can find fast ethernet hardware that'll go toe to toe with infiniband on latency and throughput.

Re: Ask HN: How would you store 10PB of data for your startup today?

#224

Post by latch »

user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

> Let's do the math.

Offers no math.

At retail, 625 16TB drives is $400000. This is about 2x the MONTHLY retail s3 pricing. Further, as we all know, AWS bandwidth pricing is absolutely bonkers (1).

I think your conclusion that S3 is "pretty good" needs a lot more math to support.

(1) https://twitter.com/eastdakota/status/1371252709836263425

Re: Ask HN: How would you store 10PB of data for your startup today?

#225

Post by oneplane »

antpls wrote: post #174
dTal wrote: post #152

Earlier quoted context omitted.

I'm no expert but I would guess it's just a fancy word for "file", as in "key-value store", as opposed to a god-proof encryption key.

In this case, wouldn't the value be multi gigabit, not the key?

Well, you'd refer to an object by its key, so while the value of the object would have the data, you could stil refer to your objects as keys, the same way we refer to files on a filesystem. It's not the file that is big, but the blocks it represents.

Re: Ask HN: How would you store 10PB of data for your startup today?

#226

Post by teitoklien »

I would go for something like Wasabi cloud storage ,

It’s api is S3 compliant.

And also I believe they have minimal cost for transferring data from S3 into wasabi , so initial setup cost should be lower too.

It should be relatively cheaper than self hosting too , when you account for hidden costs that comes with self hosting , related to managing additional employees , having protocols in place for recovering from faults , expanding the storage as you go , maintaining existing infrastructure , etc.

You can compare the prices with respect to S3 at

(https://wasabi.com/cloud-storage-pricing/#cost-estimates)

Re: Ask HN: How would you store 10PB of data for your startup today?

#227

Post by _nickwhite »

latch wrote: post #224
user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

> Let's do the math. Offers no math. At retail, 625 16TB drives is $400000. This is about 2x the MONTHLY retail s3 pricing. Further, as we all know, AWS bandwidth pricing is absolutely bonkers (1). I think your conclusion that S3 is "pretty good" needs a lot more math to support. (1) https://twitter.com/eastdakota/status/1371252709836263425

The math should also include the price of the staff who babysit 625 spinning metal disks, who likely drive to a data center multiple times a week to swap failed drives. I shudder to think if this job fell in my lap!

Re: Ask HN: How would you store 10PB of data for your startup today?

#228

Post by epistasis »

kfrzcode wrote: post #149
epistasis wrote: post #57

If you have good sysadmin/devops types, this is a few racks of storage in a datacenter. Ceph is pretty good at managing something this size, and offers an S3 interface to the data (with a few quirks). We were mostly storing massive keys that were many gigabytes, so if you have smaller keys, so I'm not sure about performance/scalding limits with smaller keys and 10PB. I'd be sure to give your team a few months to buil…

Totally out-of-band for this thread, but... what are the uses for a multi-gigabyte key?! I'm clearly unaware of some cool tech, any key words I can search?

When I say "key" I mean the blob that gets stored, but I may be misremembering or misusing S3 terms... it was large amounts of DNA sequencing data, and one of the first tasks was to add S3 support for indexed reads to our internal HTSlib fork, and since then somebody else's implementation has been added to the library. In any case, I quickly forgot about most of the details of S3 when I no longer had to deal with it directly...

Re: Ask HN: How would you store 10PB of data for your startup today?

#230

Post by throwaway823882 »

derefr wrote: post #163
lamontcg wrote: post #133

Earlier quoted context omitted.

> as long as you’ve got a capable team looking after it who chooses standard and robust software. And cheap. If you put people in charge who are looking for ways of expanding their empire and budget through spending money on EMC/VMWare/Oracle/etc/etc then you can quickly wind up spending a lot more money. Simplistic network designs, simplistic server designs, simplistic storage designs with mostly open source softwar…

> fire anyone who said words like SAN Is EBS not, itself, a SAN?

No, EBS is reliable.
Post reply on HN