21–30 of 111 posts
Cool project and kudos for open sourcing it. Noteworthy limitation: > TernFS should not be used for tiny files — our median file size is 2MB.
I initially developed it for a usecase where I needed to store billions of tiny files, and it just requires a single s3 bucket as infrastructure.
Earlier quoted context omitted.
I'm not fully up to date since we looked into this a few years ago, at the time the CERN deployments of Ceph were cited as particularly large examples and they topped out at ~30PB. Also note that when I say "single deployment" I mean that the full storage capacity is not subdivided in any way (i.e. there are no "zones" or "realms" or similar concepts). We wanted this to be the case after experiencing situations where…
There are much larger Ceph clusters, but they are enterprise owned and not really publicly talked about. Sadly I can’t share what I personally worked on.
How does TernFS compare to CephFS and why not CephFS, since it is also tested for the multiple Petabyte range?
seems like a colossusly nice design.
Cool project and kudos for open sourcing it. Noteworthy limitation: > TernFS should not be used for tiny files — our median file size is 2MB.
I have worked on exabyte-scale storage engines. There is a good engineering reason for this type of limitation. If you had 1 KiB average file size then you have quadrillions of metadata objects to quickly search and manage with fine-granularity. The kinds of operations and coordination you need to do with metadata is difficult to achieve reliably when the metadata structure itself is many PB in size. There are intere…
What are the biggest challenges in scaling metadata from a trillion to a quadrillion objects?
GPLv2-or-later, in case you were wondering https://github.com/XTXMarkets/ternfs/blob/7a4e466ac655117d24...
TernFS is Free Software. The default license for TernFS is GPL-2.0-or-later.
The protocol definitions (go/msgs/), protocol generator (go/bincodegen/) and client library (go/client/, go/core/) are licensed under Apache-2.0 with the LLVM-exception. This license combination is both permissive (similar to MIT or BSD licenses) as well as compatible with all GPL licenses. We have done this to allow people to build their own proprietary client libraries while ensuring we can also freely incorporate them into the GPL v2 licensed Linux kernel.
It'd be helpful to have a couple of usage examples that illustrate common operations, like creating a file or finding and reading one, right after the high-level overview section. Just to get an idea what happens at the service level in these cases.
How does TernFS compare to CephFS and why not CephFS, since it is also tested for the multiple Petabyte range?
(Disclaimer: I'm one of the authors of TernFS and while we evaluated Ceph I am not intimately familiar with it) Main factors: * Ceph stores both metadata and file contents using the same object store (RADOS). TernFS uses a specialized database for metadata which takes advantage of various properties of our datasets (immutable files, few moves between directories, etc.). * While Ceph is capable of storing PBs, we curr…