Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

141–150 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#141

Post by skynet-9000 »

spookthesunset wrote: post #64
daper wrote: post #58

Earlier quoted context omitted.

10x Supermicro SSG-6049P-E1CR60H servers (60 x 3,5" HDD in 4U enclosure) - $5k each 600x WESTERN DIGITAL Ultrastar DC HC550 18TB (10800PB in total) - $500 each ~$350k in hardware, up to 20kW energy consumption, should fit in two rack towers. You can host it for about $1.5k somewhere. All assuming no redundancy :)

Don’t forget labor. You need to find talent to manage your little data center. And deal with it when it shits the bed at 4:12am on Christmas morning. So toss in at least one SRE type person. Say $200k/year. Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one.... Since redundancy is a thing, double that $350k. And 10pb is what they…

10 petabytes at AWS is $210,000 per month just for storage (even excluding AWS's very high egress and transaction pricing), so even $1M (which seems like a high estimate indeed) would be amortized in less than six months.

Also, the hardware can be depreciated, which reduces its net (of taxes) cost dramatically over time.

Five years (probably the useful life of the equipment in general) of $210,000 per month is $12.6M. That's a lot of savings.

Re: Ask HN: How would you store 10PB of data for your startup today?

#143

Post by user5994461 »

skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

You realize you can't fit 10 appliances of 4U in a rack? (A rack is 42U)

There's network equipment and power equipment that requires space in the rack. There's power limitations and weight limitations on the rack that prevents to fill it to the brim.

Re: Ask HN: How would you store 10PB of data for your startup today?

#144

Post by cricalix »

skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

The thing about fitting everything in one rack, potentially, is vibration. There have been several studies into drive performance degredation from vibration, and there's noticeable impact in some scenarios. The Open Compute "Knox" design as used by Facebook spins drives up when needed, and then back down, though whether that's for vibration impact, I don't know (their cold storage use [0]).

0: https://datacenterfrontier.com/inside-facebooks-blu-ray-cold...

https://www.dtc.umn.edu/publications/reports/2005_08.pdf

https://digitalcommons.mtu.edu/cgi/viewcontent.cgi?article=1...

Re: Ask HN: How would you store 10PB of data for your startup today?

#145

Post by ignoramous »

In cloud:

Wasabi's Reserved Capacity Storage is likely to be the cheapest: https://wasabi.com/rcs/

If you front it with Cloudflare, egress would be close to free given both these companies are part of the Bandwidth Alliance: https://www.cloudflare.com/bandwidth-alliance/

Cloudflare has an images product in closed beta, but that is likely unnecessary and probably expensive for your usecase: https://blog.cloudflare.com/announcing-cloudflare-images-bet...

--

If you're curious still, take a look at Facebook's F4 (generic blob store) and Haystack (for IO bound image workloads) designs: https://archive.is/49GUM

Re: Ask HN: How would you store 10PB of data for your startup today?

#146

Post by canucker2016 »

leetrout wrote: post #140
skynet-9000 wrote: post #128

Earlier quoted context omitted.

First sentence on the linked page: "With AWS Snowball, you pay only for your use of the device and for data transfer out of AWS. "

Wow that’s up to $500,000 just to export 10PB (depending on region).

According to https://aws.amazon.com/snowball/pricing/, egress fees depends on the region, which can range from $0.03/GB (North America & parts of Europe) to $0.05/GB (parts of Asia and Africa).

So US$300K to US$500K for egress fees + cost of Snowball devices.

The major downside of Snowball in this export case is the size limit of 80TB per device - from https://aws.amazon.com/snowball/features/ :

"Snowball Edge Storage Optimized provides 80 TB of HDD capacity for block volumes and Amazon S3-compatible object storage, and 1 TB of SATA SSD for block volumes."

That'd be around 125 Snowball devices to get 10PB out.

If OP actually has 10PB on S3 currently, the OP may want to fallback to leaving the existing data on S3 and accessing new data in the new location.

Re: Ask HN: How would you store 10PB of data for your startup today?

#147

Post by thehappypm »

oneplane wrote: post #138

If I were in your shoes I'd still host it on AWS, unless your shoes have a problem with the AWS bill, but then you run into other problems: - Paying for physical space and facilities - Paying people to maintain it - Paying for DRP/BCP - Paying periodically since it doesn't last forever so it'll need replacements But if you were to have to move out of AWS but Azure and GCP aren't options, you can do: Ceph and HDDs. Du…

Strong plus-one here. Rolling your own basically means you will need an entire brand new business function to keep the lights on. That's something your entire company is going to have to adapt to. New staff, new ways of thinking about data, new problems the C-suite needs to consider. The opportunity cost alone can be immense here, since your engineers will need to spend their time working on rote data storage and not business problems.

Re: Ask HN: How would you store 10PB of data for your startup today?

#148

Post by DSingularity »

Amazing how one post will tell you that, at your scale, S3 is stupid and other posts will tell you that at your not-small-enough-and-yet not-big-enough scale S3 is the only option. I say stick with cloud. If cost is an issue go negotiate a better contract — GCP will probably give you a nice discount. Setting up a highly available service at that scale is not a walk in the park. Can you afford the distractions from your primary app while you figure it out?

Re: Ask HN: How would you store 10PB of data for your startup today?

#149

Post by kfrzcode »

epistasis wrote: post #57

If you have good sysadmin/devops types, this is a few racks of storage in a datacenter. Ceph is pretty good at managing something this size, and offers an S3 interface to the data (with a few quirks). We were mostly storing massive keys that were many gigabytes, so if you have smaller keys, so I'm not sure about performance/scalding limits with smaller keys and 10PB. I'd be sure to give your team a few months to buil…

Totally out-of-band for this thread, but... what are the uses for a multi-gigabyte key?! I'm clearly unaware of some cool tech, any key words I can search?
Post reply on HN