can we replicate something similar to this for homelab ?
garage
How AWS S3 serves 1 petabyte per second on top of slow HDDs
41–50 of 172 posts
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#42Earlier quoted context omitted.
Woah buddy, I worked with Andy for years and this is not my experience. Moving a large product like S3 around is really, really difficult, and I've always thought highly of Andy's ability to: (a) predict where he thought the product should go, (b) come up with novel ways of getting there, and (c) trimming down the product to get something in the hands of customers. Also, did you create this account for the express pu…
[flagged]
Well, the claim in question is not exactly a rebuttal of the original commenter's point despite the negative tone so I'd cut him some slack
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#43> tens of millions of disks If we assume enterprise HDDs in the double digit TB range then one can estimate that the total S3 storage volume of AWS is in the triple digit Exabyte range. That's propably the biggest storage system on planet earth.
A certain data center in Utah might top that, assuming that they have upgraded their hardware since 2013.
https://www.forbes.com/sites/kashmirhill/2013/07/24/blueprin...
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#44Earlier quoted context omitted.
Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…
It is probably worth noting that most of the listed storage systems (including S3) are designed to scale not only in hard drives, but horizontally across many servers in a distributed system. They really are not optimized for a single storage node use case. There are also other things to consider that can limit performance, like what does the storage back plane look like for those 80 HDDs, and how much throughput can…
It's a very beefy server with 4 NVMe and 20 HDD bays + a 60-drive external enclosure, 2 enterprise grade HBA cards set to multipath round-robin mode, even with 80 drives it's nowhere near the data path saturation point.
The link is a 10G 9K MTU connection, the server is only accessed via that local link.
Essentially, the drives being HDD are the only real bottleneck (besides the obvious single-node scenario).
At the moment, all writes are buffered into the NVMes via OpenCAS write-through cache, so the writes are very snappy and are pretty much ingested at the rate I can throw data at it. But the read/delete operations require at least a metadata read, and due to the very high number of small (most even empty) objects they take a lot more time than I would like.
I'm willing to sacrifice the write-through cache benefits (the write performance is actually an overkill for my use case), in order to make it a little more balanced for better List/Read/DeleteObject operations performance.
On paper, most "real" writes will be sequential data, so writing that directly to the HDDs should be fine, while metadata write operations will be handled exclusively by the flash storage, thus also taking care of the empty/small objects problem.
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#45Earlier quoted context omitted.
I always assumed the really slow tiers were tape.
Not even the higher tiers of Glacier were tape afaict (at least when it was first created), just the observation that hard drives are much bigger than you can reasonably access in useful time.
In the early days when there were articles speculating on what Glacier was backed by, it was actually on crusty old S3 gear (and at the very beginning, it was just on S3 itself as a wrapper and a hand wavy price discount, eating the costs to get people to buy in to the idea!). Later on (2018 or so) they began moving to a home grown tape-based solution (at least for some tiers).
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#46Does anyone know what is the technology stack of S3? Monolith or multiple services? I assume would have lots of queues, caches and long running workers.
Microservices for days.
I worked on lifecycle ~5 years ago and just the Standard -> Glacier transition path involved no fewer than 7 microservices.
Just determining which of the 400 trillion keys are eligible for a lifecycle action (comparing each object's metadata against the lifecycle policy on the bucket) is a massive big data job.
Always was a fun oncall when some bucket added a lifecycle rule that queued 1PB+ of data for transition or deletion on the same day. At the time our queuing had become good enough to handle these queues gracefully but our alarming hadn't figured out how to differentiate between the backlog for a single customer with a huge job and the whole system failing to process quickly enough. IIRC this was being fixed as I left.
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#47Earlier quoted context omitted.
It is probably worth noting that most of the listed storage systems (including S3) are designed to scale not only in hard drives, but horizontally across many servers in a distributed system. They really are not optimized for a single storage node use case. There are also other things to consider that can limit performance, like what does the storage back plane look like for those 80 HDDs, and how much throughput can…
It's a very beefy server with 4 NVMe and 20 HDD bays + a 60-drive external enclosure, 2 enterprise grade HBA cards set to multipath round-robin mode, even with 80 drives it's nowhere near the data path saturation point. The link is a 10G 9K MTU connection, the server is only accessed via that local link. Essentially, the drives being HDD are the only real bottleneck (besides the obvious single-node scenario). At the…
Hope you don't have expectations (over the long run) for high availability. At some point that server will come down (planned or unplanned).
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#48Earlier quoted context omitted.
It's a very beefy server with 4 NVMe and 20 HDD bays + a 60-drive external enclosure, 2 enterprise grade HBA cards set to multipath round-robin mode, even with 80 drives it's nowhere near the data path saturation point. The link is a 10G 9K MTU connection, the server is only accessed via that local link. Essentially, the drives being HDD are the only real bottleneck (besides the obvious single-node scenario). At the…
Hope you don't have expectations (over the long run) for high availability. At some point that server will come down (planned or unplanned).
For sure, there is zero expectations for any kind of hardware downtime tolerance, it's a secondary backup storage cobbled together from leftovers over many years :)
For software, at least with MinIO it's possible to do rolling updates/restarts since the 5 instances in docker-compose are enough for proper write quorum even with any single instance down.
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#49Earlier quoted context omitted.
Lustre and ZFS can do similar speeds. However, if you need high IOPS, you need flash on MDS for Lustre and some Log SSDs (esp. dedicated write and read ones) for ZFS.
Thanks, but I forgot to specify that I'm interested in S3-compatible servers only. Basically, I have a single big server with 80 high-capacity HDDs and 4 high-endurance NVMes, and it's the S3 endpoint that gets a lot of writes. So yes, for now my best candidate is ZFS + Garage, this way I can get away with using replica=1 and rely on ZFS RAIDz for data safety, and the NVMEs can get sliced and diced to act as the fast…
I'm working on something that might be suited for this use-case at https://github.com/uroni/hs5 (not ready for production yet).
It would still need a resilience/cache layer like ZFS, though.
Re: How AWS S3 serves 1 petabyte per second on top of slow HDDs
#50I enjoyed this article but I think the answer to the headline is obvious: parallelism
That's like saying "how to get to the moon is obvious: traveling"