Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

231–240 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#233

Post by latch »

_nickwhite wrote: post #227
latch wrote: post #224

Earlier quoted context omitted.

> Let's do the math. Offers no math. At retail, 625 16TB drives is $400000. This is about 2x the MONTHLY retail s3 pricing. Further, as we all know, AWS bandwidth pricing is absolutely bonkers (1). I think your conclusion that S3 is "pretty good" needs a lot more math to support. (1) https://twitter.com/eastdakota/status/1371252709836263425

The math should also include the price of the staff who babysit 625 spinning metal disks, who likely drive to a data center multiple times a week to swap failed drives. I shudder to think if this job fell in my lap!

Sure, but you actually have to go through the steps. The very first step, back of the napkin, indicates savings. More than enough to warrant a more detailed evaluation. Then you can start considering the more complicated factors (redundancy, staffing, power, ...).

My response was in the context of someone who didn't do any of that.

Also, my tongue-in-cheek response to you is: the price will be offset by the SRE engineers who were babysitting your AWS setup that you'll no longer need. (More seriously, I don't think finding quality sysadmins who enjoy this stuff is particularly harder than finding quality roles for any other tech positions).

Re: Ask HN: How would you store 10PB of data for your startup today?

#235

Post by fakedang »

pokler wrote: post #159
cricalix wrote: post #144

Earlier quoted context omitted.

The thing about fitting everything in one rack, potentially, is vibration. There have been several studies into drive performance degredation from vibration, and there's noticeable impact in some scenarios. The Open Compute "Knox" design as used by Facebook spins drives up when needed, and then back down, though whether that's for vibration impact, I don't know (their cold storage use [0]). 0: https://datacenterfront…

Here is Brendan Gregg showing how vibrations can affect disk latency: https://www.youtube.com/watch?v=tDacjrSCeq4

I'm an absolute noob here, but is using SSD racks for storage a feasible option cost wise and for this issue in particular?

Re: Ask HN: How would you store 10PB of data for your startup today?

#236

Post by aynsof »

tootie wrote: post #203
epistasis wrote: post #57

If you have good sysadmin/devops types, this is a few racks of storage in a datacenter. Ceph is pretty good at managing something this size, and offers an S3 interface to the data (with a few quirks). We were mostly storing massive keys that were many gigabytes, so if you have smaller keys, so I'm not sure about performance/scalding limits with smaller keys and 10PB. I'd be sure to give your team a few months to buil…

This is outside my domain and I don't know how the pricing works out, but AWS Outpost will sell you a physical rack that is fully S3 compatible and redundant to cloud.

The pricing would be prohibitive, I reckon. S3 on Outposts is $0.1/GB/mo, whereas the S3 single zone IA that OP is using as a baseline is $0.01/GB/mo - an order of magnitude less. (Prices are based on us-east-1.)

Re: Ask HN: How would you store 10PB of data for your startup today?

#237

Post by derefr »

jauer wrote: post #210
derefr wrote: post #166

Earlier quoted context omitted.

None of those are start-ups , though. They've either IPOed (CloudFlare, Zoom) or been acquired by publicly-traded companies (WhatsApp). A startup is a company that might still need to pivot to find its final business model, potentially shedding its entire existing infrastructure base in the process. Start-ups are why IaaS providers don't default to instance reservations — because, as a startup, you might suddenly rea…

WhatsApp ran on bare metal in SoftLayer prior to (and well after) being acquired by FB. CloudFlare went well beyond leasing servers and built their own POPs with network etc prior to IPO. Much of what they built wouldn't have made economic sense with AWS tax.

I didn't mean to imply that IPOing is the point at which a start-up becomes a not-start-up. None of these three were a start-up for quite a few years before their IPO, either.

Re: Ask HN: How would you store 10PB of data for your startup today?

#238

Post by davgoldin »

My previous startup (~2014) had a similar problem: PBs of data, with millions of mixed clients accessing it at close to real-time speeds. The biggest difference is that we needed to do real-time processing before delivering the content. We needed storage capacity balanced with CPU and RAM.

We ended up buying lots of Supermicro's ultra dense servers [1]. That's a 3U box, containing 24 servers that are interconnected with internal switches (think: 1 box is a self-contained mini cloud). Each server has (cheap config) 1 CPU 4 Xeon cores, 32GB ram, 4TB disk.

Those were bought & hosted in China, and IIRC price tag was around $20k USD per box. That's 96TB per 3U, or >1.2PB and ~$200k per rack. We had a lot of racks in multiple datacenters. These days capacity can be much larger, e.g.: 6TB disk, 144TB per 3U and >1.8PB per rack.

We've tried Ceph, GlusterFS, HDFS, even early versions of Citus, and pretty much everything that existed and was maintain during that time. We eventually settled on Cassandra. It required 2 people to maintain the software, and 1 for the hardware.

Today, I would have done the same hardware setup, mainly because I haven't had 1 Supermicro component fail on me since I bought them first in early 2000s. Cassandra would've been replaced by FoundationDB. I've been using FoundationBD for awhile now, and it just works: zero maintenance, incredible speeds, multi datacenter replication, etc.

Alternatively, if I needed storage without processing, but with fast access, I'd probably go with Supermicro's 4U 90 bay pods [2]. That'd be 90*16TB, 1.4PB in 4U, or ~14PB per rack. And FoudnationDB, no doubt.

As a fun aside: back then, we also tried Kinetic Ethernet Attached Storage [3]. Great idea but what a pain in the rear it was. We did however have a very early access device. No idea if it's still in production or not.

[1] https://www.supermicro.com/en/products/system/3U/5038/SYS-50...

[2] https://www.supermicro.com/en/products/system/4U/6048/SSG-60...

[3] https://www.supermicro.com/products/nfo/files/storage/d_SSG-...

Re: Ask HN: How would you store 10PB of data for your startup today?

#239

Post by hpcjoe »

skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

After a certain size, AWS et al simply don't make sense, unless you have infinitely deep pockets. For storage that you pull from, AWS et al charge bandwidth costs. These costs are non-trivial for non-trivial IO. I worked up financial operational models for one of my previous employers, when were were looking at costs of remaining on S3 and using it, versus rolling it into our own DCs. The download costs, the DC space, staff, etc. was far less per year (and the download cost is a 1 time cost) than the cold storage costs.

Up to about 1PB with infrequent use, AWS et al might be better. When you look at 10-100PB and beyond (we were at 500PB usable or so last I remembered) the costs are strongly biased towards in-house (or in-DC) vs cloud. That is, unless you have infinitely deep pockets.

Post reply on HN