Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

171–180 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#171

Post by user5994461 »

Dylan16807 wrote: post #93
spookthesunset wrote: post #64

Earlier quoted context omitted.

Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning. So toss in at least one SRE type person. Say $200k/year. Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... Since redundancy is a thing, double that $350k. And 10pb is what they…

> Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... This person's entire job is managing a few racks of hard drives? How often do you think they're actually going to get called in? > Since redundancy is a thing, double that $350k. True, but you can do redundancy for cheaper with parity or tape. > And 10pb is what they have no…

>>> This person's entire job is managing a few racks of hard drives? How often do you think they're actually going to get called in?

How about every day?

Quick guess how often disks need to be replaced when there are thousands of them. ;)

Re: Ask HN: How would you store 10PB of data for your startup today?

#172

Post by louwrentius »

Ceph is a beast and will require at least 2-3 technicians with intricate Ceph knowledge to run multiple (!) Ceph clusters in a business continuity responsible manner.

Because you must be able to deal with Ceph quirks.

If you can shard your data over multiple independent stand-alone ZFS boxes, that would be much simpler and more robust. But it might not scale like Ceph.

Re: Ask HN: How would you store 10PB of data for your startup today?

#173

Post by louwrentius »

user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

Very good advice!

Re: Ask HN: How would you store 10PB of data for your startup today?

#174

Post by antpls »

dTal wrote: post #152
kfrzcode wrote: post #149

Earlier quoted context omitted.

Totally out-of-band for this thread, but... what are the uses for a multi-gigabyte key?! I'm clearly unaware of some cool tech, any key words I can search?

I'm no expert but I would guess it's just a fancy word for "file", as in "key-value store", as opposed to a god-proof encryption key.

In this case, wouldn't the value be multi gigabit, not the key?

Re: Ask HN: How would you store 10PB of data for your startup today?

#175

Post by louwrentius »

philippb wrote: post #164

i just wanted to thank everyone for taking the time to reply. This has been way better input than I thought it would turn out.

I hope you will share your decision with us if you can, would be interesting to understand.

Good luck.

Re: Ask HN: How would you store 10PB of data for your startup today?

#176

Post by dividedbyzero »

monstrado wrote: post #139
pmlnr wrote: post #90

Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed servic…

MinIO is an option as well and would allow you to transition from testing in S3 to your own MinIO cluster seamlessly.

Does it scale that far?

Re: Ask HN: How would you store 10PB of data for your startup today?

#177

Post by crescentfresh »

user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

> * To store 10+TB of data.

> * You need 15 TB of storage (running at 66% capacity)

> * You need 30 TB of raw disks (twice for redundancy).

Did you mean PB?

Re: Ask HN: How would you store 10PB of data for your startup today?

#178

Post by user5994461 »

crescentfresh wrote: post #177
user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

> * To store 10+TB of data. > * You need 15 TB of storage (running at 66% capacity) > * You need 30 TB of raw disks (twice for redundancy). Did you mean PB?

Corrected.

Re: Ask HN: How would you store 10PB of data for your startup today?

#179

Post by pkb »

1) For hardware you want cheap, expendable, bare metal. Look up posts about how Google built their own servers for reference. 2) For RAID, go with software only RAID. You will sidestep problems caused by hardware RAID controllers having custom data format each (i.e. non-swapable for different model/make). 3) For filesystem, look for OpenAFS. CERN is using OpenAFS to store petabytes of data from LHC. 4) For operating system, look at Debian. Coupled with FAI (fully automated installation), it will enable you to deploy multiple servers in an automated way, to host your files.

Re: Ask HN: How would you store 10PB of data for your startup today?

#180

Post by hn_throwaway_99 »

canucker2016 wrote: post #165
nicoburns wrote: post #156

Earlier quoted context omitted.

There's also the snowmobile https://aws.amazon.com/snowmobile/

The AWS Snowmobile pages only talk about migrating INTO AWS, not OUT OF. from https://aws.amazon.com/snowmobile/ : AWS Snowmobile is an Exabyte-scale data transfer service used to move extremely large amounts of data to AWS. You can transfer up to 100PB per Snowmobile, a 45-foot long ruggedized shipping container, pulled by a semi-trailer truck. Snowmobile makes it easy to move massive volumes of data to the cloud, i…

I mean, the title on the snowmobile page says:

> Migrate or transport exabyte-scale data sets into and out of AWS

Post reply on HN