Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

291–300 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#291

Post by jauer »

dilyevsky wrote: post #290
jedberg wrote: post #216

Earlier quoted context omitted.

I've put 39U of drives in a rack before. You only need 1U for a network switch, and you can get power that attaches vertically to the back, so it doesn't take up any space. If you have a cabinet with rack in front and back and all the servers have rails, the weight shouldn't be an issue. The biggest issue will be cooling depending on how hot your servers run. Specifically, it was a rack full of Xserve RAIDs, which ar…

You run a single network switch for a rack full of drives to the brim?

Sure. A single rack is a common failure domain so you make sure to replicate across racks.

E.g. Dunno about anyone else, but Facebook racks (generally) have a single switch.

Re: Ask HN: How would you store 10PB of data for your startup today?

#292

Post by aledalgrande »

maestroia wrote: post #196

There are four hidden costs which not many have touched upon. 1) Staff You'll need at least one, maybe two, to build, operate, and maintain any self-hosted solution. A quick peek on Glassdoor and Salary show the unloaded salary for a Storage Engineer runs $92,000-130,000 US. Multiply by 1.25-1.4 for loaded cost of an employee (things like FICA, insurance, laptop, facilities, etc). Storage Administrators run lower, bu…

I am 100% agreeing with this, especially cash flow for a startup, it's going to be harder to manage. I think S3 is still the answer.

Re: Ask HN: How would you store 10PB of data for your startup today?

#293

Post by dilyevsky »

jauer wrote: post #291
dilyevsky wrote: post #290

Earlier quoted context omitted.

You run a single network switch for a rack full of drives to the brim?

Sure. A single rack is a common failure domain so you make sure to replicate across racks. E.g. Dunno about anyone else, but Facebook racks (generally) have a single switch.

That seems rather unnecessary risk. Sure you stripe across the racks but another tor switch in mlag configuration is a minuscule expense compared to the costs involved here

Re: Ask HN: How would you store 10PB of data for your startup today?

#294

Post by unnouinceput »

zmmmmm wrote: post #194

My only comment is that I have a hard time reconciling these two statements: > downloaded in the background to a mobile phone and > but less than 1000ms required I'm struggling to think of what kind of application needs data access in the background with latency of less than 1000ms. That would normally be for interactive use of some kind. Getting to 1 min access time would get you into the S3 glacier territory ... yo…

TikTok is most obvious example.

Re: Ask HN: How would you store 10PB of data for your startup today?

#295

Post by qt_scientist »

tw04 wrote: post #114

I should preface this with: I read the question as you want something on-premises/in a colo. If you're talking hosted S3 by someone other than Amazon that's a different story. It probably depends on if you are tied at the hip to other AWS services. If you are, then you're kind of stuck. The ingress/egress traffic will kill you doing anything with that data anywhere else. If you aren't, the major players for on-prem S…

[Disclaimer: I work on Quantum ActiveScale, an on-prem S3 system that fits this list]

Yup this is why these vendors exist. You’re definitely not alone, cloud repatriation is a ‘thing’.

These vendors have replaced that Ceph or Minio expert others in this thread said you’d need to budget for, by software. The system detects dead/degrading disks and automatically evicts them and rebuilds that chunk of the erasure code on another disk. Every few months you go in and hot swap the bad disks = the ones with a blinking led. Also Prometheus metrics, alerts in case of issues,... You don’t need a storage admin babysitting this.

1 rack of 4u90 enclosures with 18TB disks is 15PB RAW so with erasure coding overhead about the 10PB usable you need today.

I’m obviously biased on which vendor to pick. Do your due diligence, f.i. on how the system does capacity expansions.

Re: Ask HN: How would you store 10PB of data for your startup today?

#296

Post by d110af5ccf »

cerved wrote: post #288
atomicity wrote: post #273

Earlier quoted context omitted.

If you have a PBs of data that you rarely access, it seems to make sense to compress it first. I've rarely seen any non-giants with PBs of data properly compressed. For example, small JSON files converted into larger, compressed parquet files will use 10-100x less space. I am not familiar with images but see no reason why encoding batches of similar images should make it hard to get similar or even better compression…

couple be wrong but I don't believe compression of batches of compressed images compresses well but it'd be very interested to here about techniques on this because I have a lot of space eaten up by timelapses myself

On the contrary, batches of images with a high degree of similarity compress _very_ well. You have to use an algorithm specifically designed for that task though. Video codecs are a real world example of such - consider that H. 265 is really compressing a stream of (potentially) completely independent frames under the hood.

I'm not sure what the state of lossless algorithms might be for that though.

Re: Ask HN: How would you store 10PB of data for your startup today?

#297

Post by lars_francke »

pmlnr wrote: post #90

Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed servic…

I'd be interested to learn more about your HDFS usage and your experience at that scale. Would you be willing to have a chat? If so, my email is in my profile.

Re: Ask HN: How would you store 10PB of data for your startup today?

#298

Post by epistasis »

philippb wrote: post #66
epistasis wrote: post #57

If you have good sysadmin/devops types, this is a few racks of storage in a datacenter. Ceph is pretty good at managing something this size, and offers an S3 interface to the data (with a few quirks). We were mostly storing massive keys that were many gigabytes, so if you have smaller keys, so I'm not sure about performance/scalding limits with smaller keys and 10PB. I'd be sure to give your team a few months to buil…

The is amazing. Thank you. I’ve been looking at Backblaze storage pods that seem to be designed for that use case. Never rented rack space. Do you remember somehow the math on how much cheaper it was or how you thought about upfront cost vs ongoing. Just order of magnitude would be great.

Upfront costs, with networking, rack and stacked, and wired, were far under $100/TB raw, around $40-$60, but this was quite a while ago and I don't know how it looks in the era of 10+TB drives. Also remember that once you are off S3 you are in the situation of doing your own backup, and the use case dictates the required availability when things fail... we didn't need anything online, but mirrored to a second site. With erasure coding, you can get by with 1.5x copies at each site or so, with a performance hit. So properly backed up with a full double, it's about 3x raw...

Opex will be power, data center rent, and internet access are hugely hugely variable. And of course, the personnel will be at least 1 full time person who's extremely competent.

Re: Ask HN: How would you store 10PB of data for your startup today?

#300

Post by dhruvrrp »

hamburga wrote: post #32

Meta-question: shouldn't there be a website dedicated specifically to reliable, crowd-sourced answers to questions like these? Does it really not exist? I'm thinking like StackShare, but you start from "What's the problem I'm trying to solve?", not "What products are big companies using?".

There is http://highscalability.com/ but you have to distinguish between PR and decent technical articles.
Post reply on HN