Live data from Hacker News

Watchman: Execute a command when something changes

github.com

41–50 of 98 posts

Re: Watchman: Execute a command when something changes

#41
post #3

Not to be confused with Facebook’s file watch daemon, also names watchman, which does the same sort of thing but is more complicated. There’s a bunch of tools that integrate Facebook’s watchman for more efficient change tracking. Advantages of Facebook’s watchman: 1. Implements efficient file system event watches on macOS and Windows 2. IPC/daemon system reduces resource use because overlapping watches/triggers don’t…

If you sign up for notifications on both github projects, you can watch the watchmen.

It’s a good day when I can get a laugh out of HN heh

Re: Watchman: Execute a command when something changes

#44
post #18
post #17

Earlier quoted context omitted.

From my experience, they break every time if you are making changes with tools that replace the file rather than change it in-place, like most text editors.

In that case, you might want to monitor the directory too, not just the file. I believe this is even how it is supposed to work.

Yeah, and then filter out the events in the directory to find out when the correct file changed, etc. That is exactly what I would want the tool to abstract for me.

Re: Watchman: Execute a command when something changes

#46
post #17
post #12

From my experience, these types of notification handlers are very fragile and tend to fail every so often. Do not use if your life depends on it.

From my experience, they break every time if you are making changes with tools that replace the file rather than change it in-place, like most text editors.

At least with inotify, ie, incron, this is just a matter of setting a less stupid watch.

You can ask the kernel to watch for several different specific kinds of fs events, either at the file or directory level.

Detecting when a file is either created, or closed from writing, is no problem at all. Ie, handles the case of writing a file normally, editing a file, and renaming a file into exitence without having opened it for writing (that happened while it had some other name you weren't watching). This would handle unlinking and recreating the same as editing or uploading.

You can watch specifically for the close event only, and even more specifically for close-from-write-mode to detect updates but ignore until the update is done. And at the same time you can also watch for creation, which handles creating a file under some other name and then renaming it into existence under the name you care about without the name you care about ever having a close-from-write event.

That would fire off your script when a filename you were watching was unlinked and re-created instead of edited in-place, and then it's up to your script to handle the case where it's a "new" file that is really just replacing an old file.

I don't even see why that should be any kind of special problem. Are some tools doing something like maintaining ooen file handles or something for every watched file??? That would break by unlink-create, but that would be insane.

Perhaps these wrappers that try to simplify things are as usual just a way to break the thing they are trying to simplify. Just misguided crap.

If the feature is baked into an ide or media server and failing to handle that case, well they don't have to fail to handle that case. The subsystem totally supplies the functionality.

For something random adhoc where you're writing your own incrontab and handler script, it's no problem at all.

Re: Watchman: Execute a command when something changes

#47
post #17
post #12

From my experience, these types of notification handlers are very fragile and tend to fail every so often. Do not use if your life depends on it.

From my experience, they break every time if you are making changes with tools that replace the file rather than change it in-place, like most text editors.

Yeah, this is difficult, though not impossible, to get right with the inotify/kqueue api. Windows and mac do provide apis for recursive directory monitoring but a cross platform tool will have to solve this problem. At a high level, the way to do it is to create an in memory representation of the file system that caches a watch handle for every file. When a deletion of a file is detected, you must create a watch on the parent directory, if there wasn't one already. Then you should be able to detect the ensuing create. To make this more concrete, the problem that a file watcher needs to solve is the problem of keeping its in memory representation of the file system consistent with the actual file system. Watch events are just a useful side effect of this process.

The other fun part is that there is often a lag between the deletion and the create in the text editor case so it is necessary to defer triggering events when a deletion is detected and wait a little while to see if there is a corresponding creation. Otherwise you may rerun your command that depends on the file before it exists.

It is possible to get like a 99+% solution to this problem without polling but it is a lot harder than what these simple tools, including entr do. The upshot is that file monitoring should be looked at as a lowest common denominator solution. A better solution is to build automatic command running into the text editor itself.

Re: Watchman: Execute a command when something changes

#48
Those considering doing something like this (such as the author) might consider using my library:

https://github.com/e-dant/watcher

It hooks into system APIs where viable. (Otherwise, it uses std::filesystem.)

It’s meant to be as or more:

- Easy to use

- Efficient

Than all similar libraries.

Re: Watchman: Execute a command when something changes

#49
I've been using fswatch to great effect. The big advantage is that it's truly cross-platform, falling back to polling if no fs notification system is available as long as stat(2) works.

Very composable as you just pipe its output to whatever you want (typically a while read do end), so exclusion is a awk/sed/perl/grep/rg away, and command can be as simple or complex as it needs to while the tool itself stays lean.

Cool features include -o to just be notified that something changed (e.g to fire up rsync), and ability to batch changes.

Also the command is a front end to libfswatch, so one can use that directly if shelling is to be foregone.

https://github.com/emcrisostomo/fswatch

Re: Watchman: Execute a command when something changes

#50
post #4

For those use-cases I use entr ( https://github.com/clibs/entr ). In what sense does watchman differ to it? Didn't see anything related to this at first glance :-)

The page linked above says:

> WARNING: This is a (possibly outdated and/or unmaintained) fork of https://github.com/eradman/entr .

Post reply on HN