10000000000B
أبي خشب وحديد
درع الفضاء
سراح الفضاء
TernFS – An exabyte scale, multi-region distributed filesystem
81–90 of 111 posts
Re: TernFS – An exabyte scale, multi-region distributed filesystem
#82Earlier quoted context omitted.
If you keep all order book changes for a large number of financial instruments volume adds up quickly.
Would that kind of data not compress like crazy? Or would they need to keep all that data hot and fast.
From just a single exchange you can reach up to 1 million messages of order book change per second
https://www.nasdaqtrader.com/snippets/inet2.html
Message Volume 1,684,103,265
Messages per Second 1,134,640
Order Volume 871,875,595
Orders per Second 581,696
Share Volume 12,814,454,760
Executions per Second 193,350
I worked in this sector and volume of data is a real challenge, no wonder you often get custom software to handle that :)
Re: TernFS – An exabyte scale, multi-region distributed filesystem
#83Earlier quoted context omitted.
Amazon FSx for Lustre is the product. They do have information on DIY with the underlying tech: https://aws.amazon.com/blogs/hpc/scaling-a-read-intensive-lo...
Thanks for the link! I had seen this, but it wasn't clear to me either how to configure the host as an nvme-of target, nor whether it would actually bypass the host CPU. The article (admittedly now 4 years old) cites single digit GB/second, while I was really hoping for something closer to the full NVME bandwidth. Maybe that's just a reflection of the time though, drives have gotten a lot faster since then. Edit: thi…
Probably something more like this? https://github.com/aws-samples/amazon-fsx-tutorial/tree/mast...
Re: TernFS – An exabyte scale, multi-region distributed filesystem
#84> There's a reason why every major tech company has developed its own distributed filesystem I haven't worked at FAANG, but is this a well-known fact? I've never heard of it. Unless they're referring to things like S3? Are these large corps running literal custom filesystem implementations?
Yes, at least Facebook and Google have distributed file services. (Facebook's was historically based on HDFS, which was based on an old Google paper.)
Re: TernFS – An exabyte scale, multi-region distributed filesystem
#85> Most of the metadata activity is contained within a single shard: > > - File creation, same-directory renames, and deletion. > - Listing directory contents. > - Getting attributes of files or directories. I guess this is a trade-off between a file system and an object store? As in S3, ListObjects() is a heavy hitter and there can be potentially billions of objects under any prefix. Scanning only on a single instanc…
It's definitely a different use case but given they haven't had to tap into their follower replicas for scale, it must be pretty efficient and lightweight. I suspect not having ACLs helps. They also cite a minimum 2MB size, so not expecting exabtyes of little bytes.
I wonder if a major difference is listing a prefix in object storage vs performing recursive listings in a file system?
Even in S3, performing very large lists over a prefix is slow and small files will always be slow to work with, so regular compaction and catching file names is usually worthwhile.
Re: TernFS – An exabyte scale, multi-region distributed filesystem
#86Earlier quoted context omitted.
Would that kind of data not compress like crazy? Or would they need to keep all that data hot and fast.
From just a single exchange you can reach up to 1 million messages of order book change per second https://www.nasdaqtrader.com/snippets/inet2.html Message Volume 1,684,103,265 Messages per Second 1,134,640 Order Volume 871,875,595 Orders per Second 581,696 Share Volume 12,814,454,760 Executions per Second 193,350 Also if you look at equity derivative products which have parameters like type call/put, strike, maturit…
Thanks for the insight!
Re: TernFS – An exabyte scale, multi-region distributed filesystem
#87A few questions if the authors are around! > Is hardware agnostic and uses TCP/IP to communicate. So no RDMA? It's very hard to make effective use of modern NVMe drives bandwidth over TCP/IP. > A logical shard is further split into five physical instances, one leader and four followers, in a typical distributed consensus setup. The distributed consensus engine is provided by a purpose-built Raft-like implementation,…
Not to mention you simply want a large distributed system implemented in multiple clouds / on prems / use cases, with battle tested procedures on node failure, replacement, expansion, contraction, backup/restore, repair/verification, install guides, an error "zoo". Not to mention a Jepsen test suite, detailed CAP tradeoff explanation, etc. There's a reason those big DFS at the FAANGs aren't really implemented anywher…
My memories are a bit sketchy, but isn't CAP worked around by the eventual consistency of Paxos/Raft/...?
Re: TernFS – An exabyte scale, multi-region distributed filesystem
#88> Most of the metadata activity is contained within a single shard: > > - File creation, same-directory renames, and deletion. > - Listing directory contents. > - Getting attributes of files or directories. I guess this is a trade-off between a file system and an object store? As in S3, ListObjects() is a heavy hitter and there can be potentially billions of objects under any prefix. Scanning only on a single instanc…
It's definitely a different use case but given they haven't had to tap into their follower replicas for scale, it must be pretty efficient and lightweight. I suspect not having ACLs helps. They also cite a minimum 2MB size, so not expecting exabtyes of little bytes. I wonder if a major difference is listing a prefix in object storage vs performing recursive listings in a file system? Even in S3, performing very large…
2MB median to be fair, so half of our files are under 2MB.
Re: TernFS – An exabyte scale, multi-region distributed filesystem
#89A few questions if the authors are around! > Is hardware agnostic and uses TCP/IP to communicate. So no RDMA? It's very hard to make effective use of modern NVMe drives bandwidth over TCP/IP. > A logical shard is further split into five physical instances, one leader and four followers, in a typical distributed consensus setup. The distributed consensus engine is provided by a purpose-built Raft-like implementation,…
Not to mention you simply want a large distributed system implemented in multiple clouds / on prems / use cases, with battle tested procedures on node failure, replacement, expansion, contraction, backup/restore, repair/verification, install guides, an error "zoo". Not to mention a Jepsen test suite, detailed CAP tradeoff explanation, etc. There's a reason those big DFS at the FAANGs aren't really implemented anywher…
DeepSeek team, which is also an HFT shop, also implemented their DFS - https://github.com/deepseek-ai/3FS
Re: TernFS – An exabyte scale, multi-region distributed filesystem
#90A few questions if the authors are around! > Is hardware agnostic and uses TCP/IP to communicate. So no RDMA? It's very hard to make effective use of modern NVMe drives bandwidth over TCP/IP. > A logical shard is further split into five physical instances, one leader and four followers, in a typical distributed consensus setup. The distributed consensus engine is provided by a purpose-built Raft-like implementation,…
> So no RDMA?
We can saturate the network interfaces of our flash boxes with our very simple Go block server, because it uses sendfile under the hood. It would be easy to switch to RDMA (it’s just a transport layer change) but right now we didn’t need to. We’ve had to make some difficult prioritisation decisions here.
PRs welcome!
> Implementing distributed consensus correctly from scratch is very hard - why not use some battle-tested implementations?
We’re used to building things like this, trading systems are giant distributed systems with shared state operating at millions of updates per second. We also cheated, right now there is no automatic failover enabled. Failures are rare and we will only enable that post-Jepsen.
If we used somebody else’s implementation we would never be able to do the multi-master stuff that we need to equalise latency for non-primary regions.
> This is not true for NFSv3 and older, it tends to be stateless (no notion of open file).
Even NFSv3 needs a duplicate request cache because requests are not idempotent. Idempotency of all requests is hard to achieve but rewarding.