At that scale I would contact AWS, Backblaze and Wasabi directly to see what improvements they can offer in terms of TCO (and potentially for a longer term contract).
Ask HN: How would you store 10PB of data for your startup today?
321–330 of 374 posts
Re: Ask HN: How would you store 10PB of data for your startup today?
#322Earlier quoted context omitted.
Disclaimer: I work for Backblaze so I'm biased and you should keep me honest. :-) > Backblaze as I mentioned in another comment will be a bad idea due to the poor S3 compatible interface Backblaze released an S3 compatible API recently: https://www.backblaze.com/b2/docs/s3_compatible_api.html We're ALWAYS curious about any issue customers see, so if there is something specifically missing you use, we both want to hea…
Good to know, how does the presigned URL work? I thought it was this - https://www.backblaze.com/b2/docs/b2_get_upload_url.html . Does it function differently from this? My guess it for a mobile app, the API backend will generate the presigned upload URL hand it off to the mobile app. But certainly don't want the mobile app to have unlimited upload for a 24hr period. So one presigned URL, one upload.
> how does the presigned URL work? I thought it was this - https://www.backblaze.com/b2/docs/b2_get_upload_url.html
If you want to use Amazon S3 APIs, you do not call ANYTHING that is documented on the Backblaze website, and you especially should not call "b2_get_upload_url" because that is a B2 native API, not an Amazon S3 API. You can always tell if you are using "B2 Native" if the call starts with "b2_" -> then that has literally nothing to do with Amazon S3 compatibility, it is the custom Backblaze protocol.
If you want to find out about Amazon S3 APIs (which you use to communicate to Backblaze's Storage Cloud or Amazon S3) then you can start here: https://docs.aws.amazon.com/general/latest/gr/signature-vers... Make sure you stay ENTIRELY on the Amazon website, and only read Amazon documentation, and use the APIs Amazon talks about (but of course you are doing all of this communicating with the Backblaze Storage Cloud backend). If any of that fails in your application, or in incompatible, PLEASE LET US KNOW!!
Re: Ask HN: How would you store 10PB of data for your startup today?
#323Earlier quoted context omitted.
Good to know, how does the presigned URL work? I thought it was this - https://www.backblaze.com/b2/docs/b2_get_upload_url.html . Does it function differently from this? My guess it for a mobile app, the API backend will generate the presigned upload URL hand it off to the mobile app. But certainly don't want the mobile app to have unlimited upload for a 24hr period. So one presigned URL, one upload.
> how does the presigned URL work? I thought it was this - https://www.backblaze.com/b2/docs/b2_get_upload_url.html If you want to use Amazon S3 APIs, you do not call ANYTHING that is documented on the Backblaze website, and you especially should not call "b2_get_upload_url" because that is a B2 native API, not an Amazon S3 API. You can always tell if you are using "B2 Native" if the call starts with "b2_" -> then th…
Got it! I thought the S3 API is a wrapper on B2 native API. Good to know
Re: Ask HN: How would you store 10PB of data for your startup today?
#324Earlier quoted context omitted.
> fire anyone who said words like SAN Is EBS not, itself, a SAN?
If you narrowly focus on the words outside of the context of what "SAN" has meant in the industry for decades now, yes it is. But no, it isn't.
Can you explain more? Because I honestly don't know enough about SANs to know the difference.
To me, a "Storage Area Network" is 1. a cluster of disk-servers, serving the role of exposing logical block-storage over a protocol like iSCSI (whether directly to client machines, or managed and dynamically allocated by hypervisor software like vSphere), where 2. machines are connected to that storage cluster over a dedicated network interface, to keep LAN/WAN packets from contending for throughput with SAN packets.
By that definition, EBS is definitely a SAN. (And technically, so is my two-drive NAS, if I configure it as an iSCSI target and then run a second switch that connects to its second network port and my workstation's second network port.)
Does "SAN" imply some specific internal architecture for the storage cluster or something?
And, if so, then what do you call the type of thing that EBS is?
Re: Ask HN: How would you store 10PB of data for your startup today?
#325Non-cloud: HPE sells their Apollo 4000[^1] line, which takes 60x3.5" drives - with 16TB drives, that's 960TB each machine, one rack of 10 of these is 9PB+ therefore, which nearly covers your 10PB needs. (We have some racks like this). They are not cheap. (Note: Quanta makes servers that can take 108x3.5" drive, but they need special deep racks.) The problem here would be the "filesystem" (read: the distributed servic…
You can also get units like this direct from Western Digital/HGST. We have a system with 3 of their 4U60 units, and they weren't all that expensive. Ordering direct from HGST, we only paid a small premium on top of the cost of the SAS drives.
This is the answer that worked for us storing petabytes a decade ago.
We collaborated with OEMs and also shared/compared notes with Backblaze on rackable mass storage for commodity drives.
Backblaze published a series of iterations of designs of multi-drive chassis, and one of the OEMs would make them for other buyers as well. If you’re doing this route, read through those for considerations and lessons learned.
Performance was > 10x better than enterprise solutions. A policy to “leave dead disks dead” aka “let them rot” as said elsewhere in this thread kept maintenance cheap.
The secret sauce part making this viable for commercial online storage hosting (we hosted video) was we used disks as JBOD with an in-house meta index with P2P health awareness to place objects redundantly across disks, chassis, racks, colocation providers, and regions.
Re: Ask HN: How would you store 10PB of data for your startup today?
#326Earlier quoted context omitted.
If the colo is far and there’s plenty of headroom, it might not justify much urgency.
That's what remote hands are for. Yes, you batch the replacements up, but this is exactly what remote hands are for.
Costs less to leave them alone, and go once a year for a trash run. Cattle not pets, no trips to the vet. Don’t waste money diagnosing / fixing.
Re: Ask HN: How would you store 10PB of data for your startup today?
#327Earlier quoted context omitted.
That's what remote hands are for. Yes, you batch the replacements up, but this is exactly what remote hands are for.
Costs less to leave them alone, and go once a year for a trash run. Cattle not pets, no trips to the vet. Don’t waste money diagnosing / fixing.
'cattle not pets' is not a valid argument when you're the owner and operator of the bare metal hardware. Do you also recommend that ISPs not replace failed fans in core and edge routers and optical transport systems? Let things with dual power supplies run for six months on one failed power supply?
Also you've clearly never interacted with cattle, sheep, goats, llamas or alpacas, which absolutely do get things like veterinary care and vaccinations. Large animal vet is a whole specialty and they spend lots of time working on animals other than horses. No trips to the vet???
Re: Ask HN: How would you store 10PB of data for your startup today?
#328Earlier quoted context omitted.
That also means that you need extra spare disks in the system, which also means extra servers, extra racks, extra power feeds, extra cooling etc. If you do a 60-disk 4U setup you'll need 1 full rack of those just to get your 10PB, then you'll need yet another one for redundancy. And then a quarter for hot spares. At that point you have single-redundancy, no file history and no scaling. Is it possible? Sure. Is this s…
You don't do redundancy that way at that scale, that's completely insane. You run ceph or beegfs or Windows Storage Server and backup to tape with a tape library. If youve got big bucks (though still peanuts compared to s3) you replicate the entire setup 1:1 at a second site.
The author doesn't want a second site. And at that scale you do redundancy at that scale within the requested parameters.
If you set your object store to be resilient to single-partition loss per object (within CAP) you effectively duplicate everything once. If you want more-than-one you get into sharding to spread the risk. We're not talking about RAID here, but about replicas or copies.
Windows Storage Server doesn't belong in a setup like this, and neither does tape since it needs to be accessible in under 1s. If higher latencies were fine the author would have been able to use something between S3 IA and Glacier. Heck, you could use cold HDD storage for that kind of access. The drives would need to spin up to collect the shards to assemble at least one replica to be able to read the file, but that's still multiple orders of magnitude faster than tape.
I have written a larger post with more numbers, and unless you seriously reduce the features you use, it's not really cheaper than S3 if you start off with no physical IT and no people to support it. It's not that it isn't possible, it's just that you need to spin up an entire business unit for it and at that point you're eating way more cost.
Regardless of the object store (or filesystem if you want to go full on legacy style), you still need at least the minimum amount of physical bits on disk to be able to store the data. And pretty much no object store supports a 1:1 logical-physical storage scale. It's almost always at least 1:1.66 in degraded mode or 1:2 in minimum operational mode.
Re: Ask HN: How would you store 10PB of data for your startup today?
#329Earlier quoted context omitted.
Depends what you’re doing I suppose. I think the three companies I mentioned (CloudFlare, Zoom and WhatsApp) are good examples of infrastructure investment as a competitive advantage.
None of those are start-ups , though. They've either IPOed (CloudFlare, Zoom) or been acquired by publicly-traded companies (WhatsApp). A startup is a company that might still need to pivot to find its final business model, potentially shedding its entire existing infrastructure base in the process. Start-ups are why IaaS providers don't default to instance reservations — because, as a startup, you might suddenly rea…
In most of these cases, the companies growth from startup to not-startup was only possible because of their infrastructure advantage. Do you think Cloudflare the startup could have offered a free tier if they had to pay Amazon $0.10 per GB that their users sent over the network?
Of course not. But the free tier was a vital component of Cloudflare's growth, first-mover advantage and wide adoption.
Re: Ask HN: How would you store 10PB of data for your startup today?
#330Earlier quoted context omitted.
You don't do redundancy that way at that scale, that's completely insane. You run ceph or beegfs or Windows Storage Server and backup to tape with a tape library. If youve got big bucks (though still peanuts compared to s3) you replicate the entire setup 1:1 at a second site.
The author doesn't want a second site. And at that scale you do redundancy at that scale within the requested parameters. If you set your object store to be resilient to single-partition loss per object (within CAP) you effectively duplicate everything once. If you want more-than-one you get into sharding to spread the risk. We're not talking about RAID here, but about replicas or copies. Windows Storage Server doesn…
>We're not talking about RAID here, but about replicas or copies.
Most distributed filesystems support some form of erasure coding. Ceph does, Minio does, HDFS does, etc. So no, you don't need to duplicate everything.