Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How would you store 10PB of data for your startup today?

news.ycombinator.com

Post reply on HN
Print view View original

261–270 of 374 posts

Re: Ask HN: How would you store 10PB of data for your startup today?

#261

Post by closeparen »

walrus01 wrote: post #201
pmlnr wrote: post #90

Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed servic…

> The problem here would be the "filesystem" (read: the distributed service): I don't have much experience with Ceph, I think at that scale you would want a ceph expert on staff as a full time salaried position. For an organization that has 10PB now and can project a growth path to 15, 20, 25PB in the future, you should talk with management about creating a vacant position for that role, and filling it. > EDIT: btw,…

If the colo is far and there’s plenty of headroom, it might not justify much urgency.

Re: Ask HN: How would you store 10PB of data for your startup today?

#262

Post by dsyrk »

msk20 wrote: post #212

I don't really know much about optimizing storage costs, But You could learn from storage giants. Example is Blackblaze storage pod 6.0 according to them it holds 0.5PB with a cost of 10k$, you will need about 20*10K$ = 200K$ + Maintenance(They also publish failure rates) , The schematics and everything is in their website and according to them they have already a supplier who provides them with such devices which yo…

fyi that 10k includes no drives.

Re: Ask HN: How would you store 10PB of data for your startup today?

#263

Post by dsyrk »

nikisweeting wrote: post #6

Backblaze B2, ingress and egress are free through Cloudflare, and it's S3 compatible. It's peanuts by comparison but I've been storing ~22TB on there for years and love it. Wasabi and Glacier would be my 2nd choices.

Wait what?! There is a way to egress from free from Backblaze B2! That’s a big deal if true.

Re: Ask HN: How would you store 10PB of data for your startup today?

#264

Post by chii »

latch wrote: post #224
user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

> Let's do the math. Offers no math. At retail, 625 16TB drives is $400000. This is about 2x the MONTHLY retail s3 pricing. Further, as we all know, AWS bandwidth pricing is absolutely bonkers (1). I think your conclusion that S3 is "pretty good" needs a lot more math to support. (1) https://twitter.com/eastdakota/status/1371252709836263425

> 625 16TB drives is $400000

how much is the real estate cost of 625 drives and associated machinery to run it?

At a guess, AWS has an operating margin of about 30%, so you can approximate their cost of hardware, bandwidth, and other fixed costs as 70% of their sticker price. As a start up, can you actually get this price to be lower? I actually dont think you can, unless your operation is very small and can be done out of a home/small office.

Re: Ask HN: How would you store 10PB of data for your startup today?

#265

Post by goliatone »

monstrado wrote: post #139
pmlnr wrote: post #90

Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed servic…

MinIO is an option as well and would allow you to transition from testing in S3 to your own MinIO cluster seamlessly.

I wonder if anyone can comment if they have experience running minio at scale. It would be a pleasant surprise if a “simple” minio cluster could handle such workload

Re: Ask HN: How would you store 10PB of data for your startup today?

#266

Post by jcalabro »

I've used Wasabi a ton in the past and it's been excellent. It's already been talked about a lot in this thread, but I haven't seen their marketing video[0] linked, and it's pretty funny so I thought I'd leave it here!

https://www.youtube.com/watch?v=P7OzyTG4fCM

Re: Ask HN: How would you store 10PB of data for your startup today?

#267

Post by mark_l_watson »

quantumofalpha wrote: post #260
user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

AWS is ridiculously expensively at their scale, both for storage and egress. But the choice is not only between that and building a staffed on-premise storage facility. You can compromise at a middle ground - rent a bunch of VPS/managed servers and let the hosting companies deal with all the nastiness of managing physical hardware and CAPEX. Cost around $1.6-2/TB/month (e.g. Hetzner's SX) for raw non-redundant storag…

I agree that considering Hetzner is a good idea. I have used them often, never any problems, and very low pricing.

Re: Ask HN: How would you store 10PB of data for your startup today?

#268

Post by lunatuna »

sethhochberg wrote: post #124
mceachen wrote: post #88

Earlier quoted context omitted.

Roughly a decade ago S3 storage pricing had a ~10x premium over self-hosted. The convenience of not having to touch any hardware is expensive.

Its also important to consider how often disks will fail when you are operating hundreds of them - its probably more often than you'd think, and if you don't have someone on staff and nearby to your colo provider you're going to pay a lot in remote hands fees. Your colo facility will almost certainly have 24/7 staff on hand who can help you with tasks like swapping disks from a pile of spares, but expect to pay $300+…

I’ve used Netapps and Isilon in the past. We didn’t change any disks, they did as part of the maintenance. Not sure how the physical security worked but they were let in by the data centre staff and did their thing. I think they came in weekly.

They whole solution wasn’t cheap though and all of these extras were baked into the cost. We were getting better than S3 costs from a per TB straight up without considering power , cooling and rack space costs. Network was significantly cheaper than AWS.

Not sure on how far these NAS’ scale but I would expect deep discounts for something of this scale.

Re: Ask HN: How would you store 10PB of data for your startup today?

#269

Post by lamontcg »

derefr wrote: post #163
lamontcg wrote: post #133

Earlier quoted context omitted.

> as long as you’ve got a capable team looking after it who chooses standard and robust software. And cheap. If you put people in charge who are looking for ways of expanding their empire and budget through spending money on EMC/VMWare/Oracle/etc/etc then you can quickly wind up spending a lot more money. Simplistic network designs, simplistic server designs, simplistic storage designs with mostly open source softwar…

> fire anyone who said words like SAN Is EBS not, itself, a SAN?

If you narrowly focus on the words outside of the context of what "SAN" has meant in the industry for decades now, yes it is. But no, it isn't.

Re: Ask HN: How would you store 10PB of data for your startup today?

#270

Post by dsyrk »

gamegoblin wrote: post #187
user5994461 wrote: post #137

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That…

FWIW you can get great redundancy with far less than 2x storage factor. e.g. Facebook uses a 10:14 erasure coding scheme[1] so they can lose up to 4 disks without losing data, and that only incurs a 1.4x storage factor. If one's data is cold enough, one can go wider than this, e.g. 50:55 or something has a 1.1x factor. Not that this fundamentally changes your analysis and other totally valid points, but the 2x bit ca…

https://en.wikipedia.org/wiki/Parchive

Basically they use par2 multi-file archives for cold storage with each archive file segment scattered across different physical locations. Always fun to see the kids rediscovering tricks from the old days.

Post reply on HN