How does this compare to something like famd(8) and if it’s a marked improvement, could the techniques used here be backported to FAM?
Show HN: Filesystem Watcher
61–70 of 80 posts
Re: Show HN: Filesystem Watcher
#62I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into…
> hook into the underlying kernel events like kqueue... I'm really surprised that this sort of functionality isn't built into OS's/filesystems. I recently had to do this for HDFS, and I finally "gave up" and polled the file system like you suggest as your first option. Event notification seems like something that ought to be a fundamental feature and is best owned by the file system itself.
> I'm really surprised that this sort of functionality isn't built into OS's/filesystems
It appears to be built into macOS [1]?
> Whenever the filesystem is changed, the kernel passes notifications via the special device file /dev/fsevents to a userspace process called fseventsd
Which I assume is what they're referring to here:
> A platform filesystem watcher for Darwin is used, but certain event properties are handled by the standard library. Namely, the event time and the path type.
Re: Show HN: Filesystem Watcher
#63Earlier quoted context omitted.
It’s difficult to get it perfectly right. There is ongoing work attempting to make it more perfect. I expect a year or two before this is complete. For now though, it does do what it says. The tests I’ve run show that it is accurate over large amounts of events and time. For under 1 million files and/or directories, it uses a near-zero amount of resources. Testing on older processors shows similarly positive results.…
What do you mean near-zero? You said that inotify doesn't work (and ned14 offers his comments about it). If you are using polling, I do not understand how your approach could be using non-zero amount of resources. Let's say you are monitoring a directory with 1 million files, how can you store the state in less than 20MB of memory (which is about the most optimistic lower bound that I can think of)? What is your secr…
Near-zero is a bit loose. It keeps a relatively compact in-memory representation. You’re about right with your estimate. Having measured just now, it’s about 30mb for 1 million directories.
The baseline Watcher’s efficiency has a wide spread. When there are many thousands of nested subdirectories, the CPU approaches the limit of the thread it’s on. Flatter directories, or many files without nested subdirectories, do not have nearly as much of an effect. I’ve seen it run on around 10 million paths on a very flat test directory.
So, near-zero is somewhat misleading. There’s a wide spread in efficiency. It was my judgement that deeply nested directory trees were far less common in practice then, so I wrote “near-zero” in the optimistic case.
It uses polling under the hood (at least, I’m sure it does. It uses whatever std::filesystem uses, which is almost certainly polling).
Re: Show HN: Filesystem Watcher
#64Earlier quoted context omitted.
> hook into the underlying kernel events like kqueue... I'm really surprised that this sort of functionality isn't built into OS's/filesystems. I recently had to do this for HDFS, and I finally "gave up" and polled the file system like you suggest as your first option. Event notification seems like something that ought to be a fundamental feature and is best owned by the file system itself.
> I'm really surprised that this sort of functionality isn't built into OS's/filesystems It appears to be built into macOS [1]? > Whenever the filesystem is changed, the kernel passes notifications via the special device file /dev/fsevents to a userspace process called fseventsd Which I assume is what they're referring to here: > A platform filesystem watcher for Darwin is used, but certain event properties are handl…
> It appears to be built into macOS [1]?
It is, but it's badly implemented and buggy. But the real problem is that there is no posix like specification for file system events so every platform does it differently. Even if every platform implementation were perfect and bug free, it is a huge pain to write wrappers for each one.
Re: Show HN: Filesystem Watcher
#65Earlier quoted context omitted.
More technically, here’s what we have: A “baseline” filesystem watcher which uses only the standard library. It has been made to beat kqueue. And it does. A platform filesystem watcher for Darwin is used, but certain event properties are handled by the standard library. Namely, the event time and the path type. A platform filesystem watcher is schedule for Windows. Work hasn’t been started. A platform filesystem watc…
What do you mean by beat kqueue? Is it faster than kqueue? Does it use less memory than kqueue? How does the baseline filesystem watcher work? If it doesn't use kqueue, does it poll the filesystem periodically and diff against an in memory representation? If yes, see my other comments. If not, I am genuinely curious what you are doing because you know something that I do not.
When I began this project, I started with kqueue. The performance was wanting and there were bugs with very large file trees.
I moved to a minimal std::filesystem-based watcher and optimized it from there.
There hasn’t been a formal head-to-head test between the two. That should be about halfway down my todo list. It’s worth revisiting more formally.
My response to this question should help here: https://news.ycombinator.com/item?id=33247155#33251437
In short, there’s no secret sauce. There’s an efficiency spread in (what I consider) edge-cases.
Every potential gain over other naive watchers implemented with kqueue is likely algorithmic. I store events in a historical map, compare differences to the current state of the file tree, prune them, and send events when they change. That’s the whole implementation: scan paths, record their attributes, check for differences in the map, and send events when they happen. I haven’t given much thought to exactly why it beats kqueue, nor are there any good tests showing by how much. (Again, this is worth doing.)
Re: Show HN: Filesystem Watcher
#66When I last tried to implement this, by far the toughest part was making sure the file that’s been newly detected is done being written to. On ntfs I couldn’t find a good technique, even last modified time was not reliable. I had to watch it for changes myself.
Is "last modified" the time of the beginning of the write?
Never even thought of that; I don’t know. I assumed it was when a write was done. Whatever that means, I don’t know either.
Re: Show HN: Filesystem Watcher
#67Earlier quoted context omitted.
Is "last modified" the time of the beginning of the write?
Never even thought of that; I don’t know. I assumed it was when a write was done. Whatever that means, I don’t know either.
Would this mean that fs event based antivirus scanners could be side-stepped by writing a payload to a file and then never closing the handler?
Re: Show HN: Filesystem Watcher
#68Earlier quoted context omitted.
What do you mean by beat kqueue? Is it faster than kqueue? Does it use less memory than kqueue? How does the baseline filesystem watcher work? If it doesn't use kqueue, does it poll the filesystem periodically and diff against an in memory representation? If yes, see my other comments. If not, I am genuinely curious what you are doing because you know something that I do not.
When I began this project, I started with kqueue. The performance was wanting and there were bugs with very large file trees. I moved to a minimal std::filesystem-based watcher and optimized it from there. There hasn’t been a formal head-to-head test between the two. That should be about halfway down my todo list. It’s worth revisiting more formally. My response to this question should help here: https://news.ycombin…
Makes sense. I have only used kqueue on macos to monitor a small number of files and I find it quite painful to use and the semantics were confusing, not sure if it is different on say freebsd.
Just as a heads up, one of the strange fsevents issues is that it fails if you register two directories where one directory is a prefix of the other. So say that you want to monitor directories $ROOT/foo and $ROOT/fo and you register an event stream first with $ROOT/foo and then $ROOT/fo, you will only receive events for paths in $ROOT/fo and no events for paths in $ROOT/foo (I just double checked that this is still the case in Monterey at least). I never bothered to report this to apple but worked around it by just registering a stream with $ROOT if I detected that one path name was a substring of another.
Re: Show HN: Filesystem Watcher
#69See also: sane - for node watchexec - rust based, static binary
For CLI usage I found that the best option was instead of watching my source directory just to watch a magic file. Then I configured my editor to touch that file when saving. This has a few benefits: 1. No need to worry about which files to watch or ignoring build outputs. 2. Works with every project with no setup. 3. Easy to trigger a re-run without actually changing a file. 4. Always runs after all files are saved…
Out of curiosity, what editor do you use and how do you make sure 4. happens when for example using ‘save all’?
Re: Show HN: Filesystem Watcher
#70Earlier quoted context omitted.
For CLI usage I found that the best option was instead of watching my source directory just to watch a magic file. Then I configured my editor to touch that file when saving. This has a few benefits: 1. No need to worry about which files to watch or ignoring build outputs. 2. Works with every project with no setup. 3. Easy to trigger a re-run without actually changing a file. 4. Always runs after all files are saved…
Out of curiosity, what editor do you use and how do you make sure 4. happens when for example using ‘save all’?
I'm currently using neovim so it is pretty trivial to add save hooks. Although the approach I am currently using is just a custom shortcut that saves all files and touches the file. This way only explicit saves by me trigger the rerun.
My current setup is documented here but it's easy to tweak to your prefered workflow. https://kevincox.ca/2022/06/14/small-tools/#w