Show HN: Filesystem Watcher

Reflex for go: https://github.com/cespare/reflex

In the early days of Linux, there was a tool that saved file info to a floppy. Then you would write protect the floppy and leave it in a drive, and the tool would periodically compare OS files with that to detect alteration. I can't for the life of me remember the name, though. It was great for hardened systems.

For CLI usage I found that the best option was instead of watching my source directory just to watch a magic file. Then I configured my editor to touch that file when saving. This has a few benefits:

1. No need to worry about which files to watch or ignoring build outputs.

2. Works with every project with no setup.

3. Easy to trigger a re-run without actually changing a file.

4. Always runs after all files are saved instead of starting after the first file is saved and racing the rest.

5. Infinitely scalable.

I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into the underlying kernel events like kqueue, inotify, fsevents, ReadDirectoryChangesW, etc and report events

Option 1 uses a lot of CPU and memory (the map storing the paths being monitored could easily grow to be tens or even hundreds of megabytes if many files are being monitored, which is often the case in large source projects). I have seen tools that use polling with a 100ms interval continuously burn 50% of cpu monitoring a modest sized directory with tens of thousands of files.

Option 2 theoretically would use less memory and little to no cpu, but in practice, the story is more complicated. If you are using an inotify or kqueue like api, you will have to store handles for all of the paths that are being monitored, which can take a significant amount of memory. On macos, the file system events are not accurate in the sense that you can't trust the type of event. It doesn't reliably distinguish between creation and modification events. So if you want to know specifically what kind of event happened, you end up back in case 1 where you have to store an in memory representation of the file system and diff against the in memory representation and the current file system state when you detect an event. For some use cases, you may not care to distinguish between creations and modifications and can get away with a lower memory, but less accurate, solution.

In my experience, getting all of this right is much more difficult than it appears at first glance. Good luck to you.

I don't know about efficient, but at least it sounds reliable. I'm at my wit's end with trying to figure out why KDE's Dolphin can't reliably watch a directory for new files, frequently (but not always) forcing me to F5 to see new files.