TernFS – an exabyte scale, multi-region distributed filesystem
1–10 of 32 posts
Re: TernFS – an exabyte scale, multi-region distributed filesystem
#2Previously: https://news.ycombinator.com/item?id=45290245
Re: TernFS – an exabyte scale, multi-region distributed filesystem
#3should post again when having 5% of the features of the other parallel file systems starting with RDMA, whereby it's not clear if this FS does even stripe that is if it is even a parallel file system
Re: TernFS – an exabyte scale, multi-region distributed filesystem
#4Great default license.
Re: TernFS – an exabyte scale, multi-region distributed filesystem
#5Great default license.
CephFS looks stable, and has diskprediction and Prometheus modules:
https://docs.ceph.com/en/quincy/cephfs/index.html
Still not completely decoupled from host roles, but seems to work for some folks. =3
Re: TernFS – an exabyte scale, multi-region distributed filesystem
#6Could anybody with applicable experience tell me how this filesystem compares in the real world to Lustre?
If it is decisively better than Lustre, I am happy to make the switch over at my sector in Argonne National Lab where we currently keep about 0.7 PB of image data and eventually intend to hold 3-5 PB once we switch over all 3 of our beamlines to using Dectris X-Ray detectors.
Contrary to what the non-computer scientists insist, we only need about 20Gb/s of throughput in either direction, so robustness and simplicity are the only concerns we have.
Re: TernFS – an exabyte scale, multi-region distributed filesystem
#7Isn't this literally what ZFS is designed for? What is ZFS lacking that this is needed.
Re: TernFS – an exabyte scale, multi-region distributed filesystem
#8Isn't this literally what ZFS is designed for? What is ZFS lacking that this is needed.
ZFS is not distributed. So probably closer to ceph or lustre. I have to admit, on my first pass through the page it failed to explain why it was better than ceph.
Re: TernFS – an exabyte scale, multi-region distributed filesystem
#9Isn't this literally what ZFS is designed for? What is ZFS lacking that this is needed.
ZFS is not distributed. So probably closer to ceph or lustre. I have to admit, on my first pass through the page it failed to explain why it was better than ceph.
Given all the good work ZFS does locally, it does make you wonder what it would take to extend the concepts of ARC caching and RAID redundancy to a distributed system, one where all the nodes are joined together by RDMA rather than ethernet; one where reliability can be taken for granted (short of a rat chewing cables).
It would make for one heck of a FreeBSD development project grant, considering how superb their ZFS and their networking stack are separately.
P.S. Glad someone pointed this out tactfully. A lot of people would have pounced on the chance to mock the poor commenter who just didn't know what he didn't know. The culture associated with software development falsely equates being opinionated with being knowledgeable, so hopefully we get a lot more people reducing the stigma of not knowing and reducing the stigma of saying "I don't know".
Re: TernFS – an exabyte scale, multi-region distributed filesystem
#10Earlier quoted context omitted.
ZFS is not distributed. So probably closer to ceph or lustre. I have to admit, on my first pass through the page it failed to explain why it was better than ceph.
Given all the good work ZFS does locally, it does make you wonder what it would take to extend the concepts of ARC caching and RAID redundancy to a distributed system, one where all the nodes are joined together by RDMA rather than ethernet; one where reliability can be taken for granted (short of a rat chewing cables). It would make for one heck of a FreeBSD development project grant, considering how superb their ZF…
Even if you were build a ZFS mega-machine with an Exabyte of storage with RDMA (the latencies of "normal" Ethernet in the datacenters would probably not be good enough), wouldn't you still have the problem that ZFS is fundamentally designed to be managed by and accessed on one machine? All data in and out of it would have to flow through that machine, which would be quite the bottleneck.