Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TernFS – an exabyte scale, multi-region distributed filesystem

xtxmarkets.com

Post reply on HN
Print view View original

11–20 of 32 posts

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#11

Post by mgerdts »

nh2 wrote: post #10
president_zippy wrote: post #9

Earlier quoted context omitted.

Given all the good work ZFS does locally, it does make you wonder what it would take to extend the concepts of ARC caching and RAID redundancy to a distributed system, one where all the nodes are joined together by RDMA rather than ethernet; one where reliability can be taken for granted (short of a rat chewing cables). It would make for one heck of a FreeBSD development project grant, considering how superb their ZF…

Even if you were build a ZFS mega-machine with an Exabyte of storage with RDMA (the latencies of "normal" Ethernet in the datacenters would probably not be good enough), wouldn't you still have the problem that ZFS is fundamentally designed to be managed by and accessed on one machine? All data in and out of it would have to flow through that machine, which would be quite the bottleneck.

If your entire system is connected via RDMA networks (rather common in HPC) I would not worry at all about latency. If you are buying NICs and switches that are capable of 100Gb or better, there’s a reasonable chance they support RoCE.

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#12

Post by mat_epice »

president_zippy wrote: post #6

Could anybody with applicable experience tell me how this filesystem compares in the real world to Lustre? If it is decisively better than Lustre, I am happy to make the switch over at my sector in Argonne National Lab where we currently keep about 0.7 PB of image data and eventually intend to hold 3-5 PB once we switch over all 3 of our beamlines to using Dectris X-Ray detectors. Contrary to what the non-computer sc…

There are several other systems I would recommend before TernFS for your environment. If you're looking at Lustre versus this in particular, Lustre has been through the wringer, and ANL/DOE has plenty of people who understand it enough to run it well and fix it when it breaks.

However, you are right. Your bandwidth needs don't really require Lustre.

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#13

Post by mgerdts »

president_zippy wrote: post #9
somat wrote: post #8

Earlier quoted context omitted.

ZFS is not distributed. So probably closer to ceph or lustre. I have to admit, on my first pass through the page it failed to explain why it was better than ceph.

Given all the good work ZFS does locally, it does make you wonder what it would take to extend the concepts of ARC caching and RAID redundancy to a distributed system, one where all the nodes are joined together by RDMA rather than ethernet; one where reliability can be taken for granted (short of a rat chewing cables). It would make for one heck of a FreeBSD development project grant, considering how superb their ZF…

This is hobby project I’ve been thinking about for quite a while. It’s way larger than a hobby project, though.

I think the key to making it horizontally scalable is to allow each writable dataset to be managed by a single node at a time. Writes would go to blocks reserved for use by a particular node, but at least some of those blocks will be on remote drives via nvmeof or similar. All writes would be treated as sync writes so another node could have lossless takeover via ZIL replay.

Read-only datasets (via property or snapshot, including clone origins) could be read directly from any node. Repair of blocks would be handled by a specific node that is responsible for that dataset.

A primary node would be responsible for managing association between nodes and datasets, including balancing load and handling failover. It would probably be responsible for metadata changes(datasets, properties, nodes, devs, etc., not posix fs metadata) and the coordination required across nodes.

I don’t feel like I have a good handle on how TXG syncs would happen, but I don’t think that is insurmountable.

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#14

Post by president_zippy »

mat_epice wrote: post #12
president_zippy wrote: post #6

Could anybody with applicable experience tell me how this filesystem compares in the real world to Lustre? If it is decisively better than Lustre, I am happy to make the switch over at my sector in Argonne National Lab where we currently keep about 0.7 PB of image data and eventually intend to hold 3-5 PB once we switch over all 3 of our beamlines to using Dectris X-Ray detectors. Contrary to what the non-computer sc…

There are several other systems I would recommend before TernFS for your environment. If you're looking at Lustre versus this in particular, Lustre has been through the wringer, and ANL/DOE has plenty of people who understand it enough to run it well and fix it when it breaks. However, you are right. Your bandwidth needs don't really require Lustre.

Seriously man, I'm asking because I don't know: which filesystems do you recommend instead? I dabbled in CephFS because our data is write-once, but helping computer illiterate research scientists at other universities and national labs retrieve their data is a lot simpler from Lustre because it's just plain-old POSIX filesystem semantics.

I'm not joking, I didn't ask this as a way to namedrop my experience and credentials (common 'round this neck o' the woods), I honestly don't know what all the much more competent organizations are doing and would really like to find out.

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#15

Post by huntaub »

president_zippy wrote: post #14
mat_epice wrote: post #12

Earlier quoted context omitted.

There are several other systems I would recommend before TernFS for your environment. If you're looking at Lustre versus this in particular, Lustre has been through the wringer, and ANL/DOE has plenty of people who understand it enough to run it well and fix it when it breaks. However, you are right. Your bandwidth needs don't really require Lustre.

Seriously man, I'm asking because I don't know: which filesystems do you recommend instead? I dabbled in CephFS because our data is write-once, but helping computer illiterate research scientists at other universities and national labs retrieve their data is a lot simpler from Lustre because it's just plain-old POSIX filesystem semantics. I'm not joking, I didn't ask this as a way to namedrop my experience and creden…

I’d be happy to chat more about your needs and try to help recommend a path forward. Feel free to shoot me an email at the address in my profile.

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#16

Post by president_zippy »

nh2 wrote: post #10
president_zippy wrote: post #9

Earlier quoted context omitted.

Given all the good work ZFS does locally, it does make you wonder what it would take to extend the concepts of ARC caching and RAID redundancy to a distributed system, one where all the nodes are joined together by RDMA rather than ethernet; one where reliability can be taken for granted (short of a rat chewing cables). It would make for one heck of a FreeBSD development project grant, considering how superb their ZF…

Even if you were build a ZFS mega-machine with an Exabyte of storage with RDMA (the latencies of "normal" Ethernet in the datacenters would probably not be good enough), wouldn't you still have the problem that ZFS is fundamentally designed to be managed by and accessed on one machine? All data in and out of it would have to flow through that machine, which would be quite the bottleneck.

Because RDMA latency is still a lot lower than disk access latency, it depends more on whether or not the control logic can be generalized to distributed scale with some simple refactoring and a few calls to access remote shared memory, or whether a full-on rewrite is less time-consuming. I don't know, and I don't pretend to know.

All I know is that the semantics of RDMA (absent experience writing code that uses RDMA) deceive me into thinking there's some possibility I could try it and not end up regretting the time spent on a proof of concept.

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#17

Post by anon-3988 »

president_zippy wrote: post #6

Could anybody with applicable experience tell me how this filesystem compares in the real world to Lustre? If it is decisively better than Lustre, I am happy to make the switch over at my sector in Argonne National Lab where we currently keep about 0.7 PB of image data and eventually intend to hold 3-5 PB once we switch over all 3 of our beamlines to using Dectris X-Ray detectors. Contrary to what the non-computer sc…

0.7PB of compressed data?

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#18

Post by x______________ »

huntaub wrote: post #15
president_zippy wrote: post #14

Earlier quoted context omitted.

Seriously man, I'm asking because I don't know: which filesystems do you recommend instead? I dabbled in CephFS because our data is write-once, but helping computer illiterate research scientists at other universities and national labs retrieve their data is a lot simpler from Lustre because it's just plain-old POSIX filesystem semantics. I'm not joking, I didn't ask this as a way to namedrop my experience and creden…

I’d be happy to chat more about your needs and try to help recommend a path forward. Feel free to shoot me an email at the address in my profile.

Is this an ad? Why can't the topic continue here as a reply to op?

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#19

Post by toast0 »

president_zippy wrote: post #6

Could anybody with applicable experience tell me how this filesystem compares in the real world to Lustre? If it is decisively better than Lustre, I am happy to make the switch over at my sector in Argonne National Lab where we currently keep about 0.7 PB of image data and eventually intend to hold 3-5 PB once we switch over all 3 of our beamlines to using Dectris X-Ray detectors. Contrary to what the non-computer sc…

If you only need 20 Gb/s, you might be able to meet your needs without an exotic distributed filesystem by just getting a single giant server with a rack full of JBODs:

Something like this [1] gets you 44 disks in 4u. You can probably fit 9 of those and a server with enough HBAs to interface with it in a 42U rack. 9x44x20TB = not quite 8 PB. Adjust for redundancy and/or larger drives. If you go with SAS drives, you can have two servers connected to the drives, with failover. Or you can setup two of these racks in different locations and mirror the data (somehow).

[1] https://www.supermicro.com/en/products/chassis/4U/847/SC847E... (as an illustration, sas jbods aka disk shelves are widely available from server vendors)

Re: TernFS – an exabyte scale, multi-region distributed filesystem

#20

Post by darkwater »

To the OP: nice karma trick posting the URL with the anchor to bypass the HN duplicates detector. Dang & co, this is a bug, it should be fixed.

I know because I stumbled on the same page following the links from the blog of the author of another post that made the frontpage yesterday (https://news.ycombinator.com/item?id=45589156), liked the TernFS concept, submitted it and got redirected to https://news.ycombinator.com/item?id=45290245

Post reply on HN