Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

201–210 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#201

Post by walrus01 »

pmlnr wrote: post #90

Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed servic…

> The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph,

I think at that scale you would want a ceph expert on staff as a full time salaried position.

For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it.

> EDIT: btw, we let the dead drives "rot": replacing them would cost more, and the failure rate is not that bad, so they stay in the machine, and we disable them in fstabs, configs, etc.

I am a huge advocate of hosting stuff yourself on bare metal you own, but this is a ridiculous statement. Any drive in that class should come with a 3 or 5 year warranty. And the manual labor and hassle time to replace one (you have hundreds of thousands of dollars of storage and no ready to go cold spares on a shelf?!?!) is infinitesimal.

Re: Ask HN: How would you store 10PB of data for your startup today?

#203

Post by tootie »

epistasis wrote: post #57

If you have good sysadmin/devops types, this is a few racks of storage in a datacenter. Ceph is pretty good at managing something this size, and offers an S3 interface to the data (with a few quirks). We were mostly storing massive keys that were many gigabytes, so if you have smaller keys, so I'm not sure about performance/scalding limits with smaller keys and 10PB. I'd be sure to give your team a few months to buil…

This is outside my domain and I don't know how the pricing works out, but AWS Outpost will sell you a physical rack that is fully S3 compatible and redundant to cloud.

Re: Ask HN: How would you store 10PB of data for your startup today?

#204

Post by pier25 »

user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

> If it's the bandwidth. You need to improve your CDN and caching layer.

What would you recommend for this?

(considering data is stored in S3)

Re: Ask HN: How would you store 10PB of data for your startup today?

#205

Post by XorNot »

I'm wondering here if this data is currently oversized? If the use case is all mobile, has your product committed to losslessly storing something or not?

While there's definitely a cross-over point where you should roll your own, the overhead costs of running a storage cluster reliably (and all the problem you don't really have to deal with because they're outsourced to AWS) mean it might be a better use of time and effort to see how much you can cut that number down by changing the parameters of your storage. The immediate savings will be much easier to justify.

Keep in mind you've also got a migration problem: getting 10PB off Amazon is not a simple, handsfree project.

Re: Ask HN: How would you store 10PB of data for your startup today?

#206

Post by pas »

dividedbyzero wrote: post #176
monstrado wrote: post #139

Earlier quoted context omitted.

MinIO is an option as well and would allow you to transition from testing in S3 to your own MinIO cluster seamlessly.

Does it scale that far?

Likely not that "gracefully". But Ceph absolutely does, and has an S3 gateway.

Re: Ask HN: How would you store 10PB of data for your startup today?

#207

Post by specktr »

more_corn wrote: post #195
pmlnr wrote: post #90

Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed servic…

Don't buy HPE gear. Qualify the gear with sample units from a few competing vendors and you'll see why.

I’ve done qualification on hpe and various competing vendors and honestly haven’t seen dramatic differences in terms of performance and failure rates. From my experience the biggest difference was with vendor support services rather than the actual hardware. I’d be curious to hear more about your qualification experience with this particular vendor if you’d be willing.

Re: Ask HN: How would you store 10PB of data for your startup today?

#208

Post by orcinus8 »

hn_throwaway_99 wrote: post #180
canucker2016 wrote: post #165

Earlier quoted context omitted.

The AWS Snowmobile pages only talk about migrating INTO AWS, not OUT OF. from https://aws.amazon.com/snowmobile/ : AWS Snowmobile is an Exabyte-scale data transfer service used to move extremely large amounts of data to AWS. You can transfer up to 100PB per Snowmobile, a 45-foot long ruggedized shipping container, pulled by a semi-trailer truck. Snowmobile makes it easy to move massive volumes of data to the cloud, i…

I mean, the title on the snowmobile page says: > Migrate or transport exabyte-scale data sets into and out of AWS

Unfortunately the header is misleading. The FAQ says:

Q: Can I export data from AWS with Snowmobile?

Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary

Re: Ask HN: How would you store 10PB of data for your startup today?

#209

Post by cdavid »

skynet-9000 wrote: post #106

At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…

[disclaimer: while I have some small experience putting things in DC, including big GPU servers, I have never been anywhere near that scale, certainly not storage]

10k $ is for a server with no hard drive. W/ 12 Gb disks, and with enough RAM, we're talking closer to 40-50k$ per server. Let's say for simplicity you're going to need to buy 15 of those, and let's say you only need to replace 2 of them per year. You need 25 over five years, that's already ~ 750k $ over 5 years.

And then you need to factor in the network equipment, the hosting in a colocation space, and if storage is your core value, you need to think about disaster recovery.

You will need at least 2 people full time on this, in the US that means minimum 2x 150k$ of costs per year: over 5 years, that's 1.5m$. If you use software-defined storage, that's likely gonna cost you much more because of the skill demand.

Altogether that's all gonna cost you much more than 500k$ over 5 years. I would say you would need at least 5x to 10x this.

Re: Ask HN: How would you store 10PB of data for your startup today?

#210

Post by jauer »

derefr wrote: post #166
TechBro8615 wrote: post #154

Earlier quoted context omitted.

Depends what you’re doing I suppose. I think the three companies I mentioned (CloudFlare, Zoom and WhatsApp) are good examples of infrastructure investment as a competitive advantage.

None of those are start-ups , though. They've either IPOed (CloudFlare, Zoom) or been acquired by publicly-traded companies (WhatsApp). A startup is a company that might still need to pivot to find its final business model, potentially shedding its entire existing infrastructure base in the process. Start-ups are why IaaS providers don't default to instance reservations — because, as a startup, you might suddenly rea…

WhatsApp ran on bare metal in SoftLayer prior to (and well after) being acquired by FB.

CloudFlare went well beyond leasing servers and built their own POPs with network etc prior to IPO. Much of what they built wouldn't have made economic sense with AWS tax.

Post reply on HN