91–100 of 111 posts
Earlier quoted context omitted.
Shameless plug: https://github.com/Barre/ZeroFS I initially developed it for a usecase where I needed to store billions of tiny files, and it just requires a single s3 bucket as infrastructure.
Interesting, can I use SeaweedFS as bucket provider?
Earlier quoted context omitted.
Not to mention you simply want a large distributed system implemented in multiple clouds / on prems / use cases, with battle tested procedures on node failure, replacement, expansion, contraction, backup/restore, repair/verification, install guides, an error "zoo". Not to mention a Jepsen test suite, detailed CAP tradeoff explanation, etc. There's a reason those big DFS at the FAANGs aren't really implemented anywher…
My memories are a bit sketchy, but isn't CAP worked around by the eventual consistency of Paxos/Raft/...?
CAP cannot be worked around. In the event of a partition, your system is either C or A, no buts. Either the losing side of the partition refuses to process writes and usually reads as well (ensuring consistency and ensuring unavailability) or it does not refuse (ensuring availability and ensuring data corruption). There are no third options.
Well, some people say the third option is to just make sure the network is 100% reliable and a partition never occurs. That's laughable.
Earlier quoted context omitted.
you really notice metadata performance (try a git checkout on EFS on AWS. loads of small files takes fucking ages) However EFS is actually pretty fast. you can get decent throughput if you're writing to just one file. but if you're trying to open 1000 1meg files to read from vs 1 1G file, it'll be much slower (unless they'd dramatically improved performance recently) Trying to have a fast globally consistent database…
We are truly spoiled by all the improvements that went into local filesystems that are lacking in network filesystems. So much of our perception of "computer is fast" is really just write-caching, read-caching, read-ahead.
In 2008 when I was a youngen, 100tb filesystem that could sustain 1-3gigabytes of streaming throughput took something like 40 racks. Huge amounts of cost and power were needed to set it up and maintain it. Any kind of random IO would kneecap the performance for everyone
Now you can have a 2u server with 100tb of NVME storage and the only bottleneck is the network adaptor! not only that but its pretty cheap too.
Earlier quoted context omitted.
My memories are a bit sketchy, but isn't CAP worked around by the eventual consistency of Paxos/Raft/...?
The protocols you mentioned are always consistent. You will know if they are not consistent because they will not make progress. Yes there's a short delay where some nodes haven't learned about the new thing yet and only know that they're about to learn the new thing, but that's not what's meant by "eventual consistency", which is when inconsistent things may happen and become consistent at some time later. In Paxos…
Thanks, I haven't looked at these problems in a while.
> In the event of a partition, your system is either C or A, no buts.
Fair enough. Raft and Paxos provide well-understood tradeoffs but not a workaround.
Earlier quoted context omitted.
If you keep all order book changes for a large number of financial instruments volume adds up quickly.
Would that kind of data not compress like crazy? Or would they need to keep all that data hot and fast.
Sounds more like an object system (immutable) with the veneer of a file system for their use cases. I sort of read the doc - sounds like data is replicated and not erasure encoded (so perhaps more expensive?). I think many people have said this, but "file systems" get a lot easier if you don't have to worry about overwrites, appends, truncates, etc. Anyway, always interesting to see what people come up with for their…
It also does not at all preclude implementing a read-write layer on top of it, for instance with a log-structured FS design. That's however the solution to a problem these people are, it seems, not having.
Earlier quoted context omitted.
This sounds like a fascinating niche piece of technical expertise I would love to hear more about. What are the biggest challenges in scaling metadata from a trillion to a quadrillion objects?
It is dependent on the intended workload but there are a few common design problems. Keep in mind that you can't just deal in the average case, you have to design for the worst possible cases of extremely skewed or pathologically biased distributions. A lot of the design work is proving worst case resource bounds under various scenarios and then proving the worst case behavior of designs intended to mitigate that. An…
Earlier quoted context omitted.
There are definitely insanely large Ceph deployments. I have seen hundreds of PBs in production myself. Also your usecase sounds like something that should be quite manageable for Ceph to handle due to limited metadata activity, which tends to be the main painpoint with CephFS.
I'm not fully up to date since we looked into this a few years ago, at the time the CERN deployments of Ceph were cited as particularly large examples and they topped out at ~30PB. Also note that when I say "single deployment" I mean that the full storage capacity is not subdivided in any way (i.e. there are no "zones" or "realms" or similar concepts). We wanted this to be the case after experiencing situations where…
It would be really interesting to see larger clusters join in on their telemetry as well.
[0] https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?orgI...