51–60 of 80 posts
Earlier quoted context omitted.
I'm sorry, I don't know what issue/10 means? Is it an e-zine or something? (apologies if this is obvious, I'm extremely tired!)
https://github.com/e-dant/watcher/issues/10
Have you looked into using eBPF for tracking file system changes at any point? (I don't mean for this project, as it's clear you're taking a particular approach that will work across platforms).
I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into…
It’s difficult to get it perfectly right. There is ongoing work attempting to make it more perfect. I expect a year or two before this is complete. For now though, it does do what it says. The tests I’ve run show that it is accurate over large amounts of events and time. For under 1 million files and/or directories, it uses a near-zero amount of resources. Testing on older processors shows similarly positive results.…
For what it's worth, in spite of ned14's comments, I have never seen inotify fail in practice (except for if it hits the os file descriptor limits in which case it does fail noisily). The tool I wrote uses inotify for linux. It is used by thousands of developers every day as part of an editor integration and there are no open issues about dropped file events.
Your time frame is probably about right. It took me about a year to work through all the edge cases.
I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into…
More technically, here’s what we have: A “baseline” filesystem watcher which uses only the standard library. It has been made to beat kqueue. And it does. A platform filesystem watcher for Darwin is used, but certain event properties are handled by the standard library. Namely, the event time and the path type. A platform filesystem watcher is schedule for Windows. Work hasn’t been started. A platform filesystem watc…
How does the baseline filesystem watcher work? If it doesn't use kqueue, does it poll the filesystem periodically and diff against an in memory representation? If yes, see my other comments. If not, I am genuinely curious what you are doing because you know something that I do not.
I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into…
I'm really surprised that this sort of functionality isn't built into OS's/filesystems. I recently had to do this for HDFS, and I finally "gave up" and polled the file system like you suggest as your first option. Event notification seems like something that ought to be a fundamental feature and is best owned by the file system itself.
"Watcher is extremely efficient. In most cases, even when scanning millions of paths, this library uses a near-zero amount of resources." Yea, maybe or maybe not and my first guess is maybe not. This needs at least some bullet points on HOW it does this so efficiently so that I'll keep looking. A blanket statement like this means "they hope it is efficient" or "They want it to be efficient" or "It's good in some scen…
If so, there are equivalent options, including systemd path units, incron, and the inotifywait utility, in addition to the C API.
The "man systemd.path" page does list explicit limitations of this kernel system call:
"Internally, path units use the inotify(7) API to monitor file systems. Due to that, it suffers by the same limitations as inotify, and for example cannot be used to monitor files or directories changed by other machines on remote NFS file systems." (Files modified by mmap() also don't trigger events.)
https://www.linuxjournal.com/content/linux-filesystem-events...
Windows busybox also has an inotifyd, which appears to do something similar.
Earlier quoted context omitted.
> instant local search still checks out :) But I checked just to be sure and no warning in Version 106.0.5249.91 (Official Build) (32-bit). Maybe its your corporate baby content web filter?
No, its a certificate issue; probably minor, but a server-side thing to attend to by the looks of it.
I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into…
> hook into the underlying kernel events like kqueue... I'm really surprised that this sort of functionality isn't built into OS's/filesystems. I recently had to do this for HDFS, and I finally "gave up" and polled the file system like you suggest as your first option. Event notification seems like something that ought to be a fundamental feature and is best owned by the file system itself.
The magic file approach described by kevincox below is probably the best way to get > 95% of the benefit with < 1% of the work.
When I last tried to implement this, by far the toughest part was making sure the file that’s been newly detected is done being written to. On ntfs I couldn’t find a good technique, even last modified time was not reliable. I had to watch it for changes myself.