In the early days of Linux, there was a tool that saved file info to a floppy. Then you would write protect the floppy and leave it in a drive, and the tool would periodically compare OS files with that to detect alteration. I can't for the life of me remember the name, though. It was great for hardened systems.
Show HN: Filesystem Watcher
41–50 of 80 posts
Re: Show HN: Filesystem Watcher
#42Any reason for making delay_ms a template parameter? A compiler should be able to optimize passing a constant as a regular function argument. And if it’s not optimized I assume a variable delay wouldn’t affect much?
No perfectly good reason. I will look into that before version 1.
Re: Show HN: Filesystem Watcher
#43forgot "written in C++" .. unless you're trying to get views, better to leave that part out.
What do you mean?
Re: Show HN: Filesystem Watcher
#44How does this work under the covers on Linux? Is it using eBPF, or is it simply an abstraction over inotify? I'm particularly interested in something like this, but which will include information about what process made the change, and which user it was running as at the time.
I’ve gone back and forth with inotify on Linux. Ned14 gave a great rundown of the ideal next steps for a best-possible implementation.
You can check out issue/10 for a full description of how it works now, why neither inotify nor our current solution is ideal, and where the project will be going next.
Re: Show HN: Filesystem Watcher
#45How does this work under the covers on Linux? Is it using eBPF, or is it simply an abstraction over inotify? I'm particularly interested in something like this, but which will include information about what process made the change, and which user it was running as at the time.
I’ve gone back and forth with inotify on Linux. Ned14 gave a great rundown of the ideal next steps for a best-possible implementation. You can check out issue/10 for a full description of how it works now, why neither inotify nor our current solution is ideal, and where the project will be going next.
I'm sorry, I don't know what issue/10 means? Is it an e-zine or something? (apologies if this is obvious, I'm extremely tired!)
Re: Show HN: Filesystem Watcher
#46I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into…
It’s difficult to get it perfectly right.
There is ongoing work attempting to make it more perfect.
I expect a year or two before this is complete.
For now though, it does do what it says. The tests I’ve run show that it is accurate over large amounts of events and time. For under 1 million files and/or directories, it uses a near-zero amount of resources. Testing on older processors shows similarly positive results.
But this is so far from perfect. This is only the groundwork. Most of the bugs have yet to be discovered. The platform support, more often than not, uses the safe “baseline” watcher in favor of accuracy.
Ned14 of Boost fame has given the project some expert advice which will help it along smoothly.
Re: Show HN: Filesystem Watcher
#47Earlier quoted context omitted.
I’ve gone back and forth with inotify on Linux. Ned14 gave a great rundown of the ideal next steps for a best-possible implementation. You can check out issue/10 for a full description of how it works now, why neither inotify nor our current solution is ideal, and where the project will be going next.
I'm sorry, I don't know what issue/10 means? Is it an e-zine or something? (apologies if this is obvious, I'm extremely tired!)
Re: Show HN: Filesystem Watcher
#48"Watcher is extremely efficient. In most cases, even when scanning millions of paths, this library uses a near-zero amount of resources." Yea, maybe or maybe not and my first guess is maybe not.
This needs at least some bullet points on HOW it does this so efficiently so that I'll keep looking. A blanket statement like this means "they hope it is efficient" or "They want it to be efficient" or "It's good in some scenarios but not others".
With those additional bits, I have a reason to dig around the source.
Re: Show HN: Filesystem Watcher
#49I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into…
More technically, here’s what we have:
A “baseline” filesystem watcher which uses only the standard library. It has been made to beat kqueue. And it does.
A platform filesystem watcher for Darwin is used, but certain event properties are handled by the standard library. Namely, the event time and the path type.
A platform filesystem watcher is schedule for Windows. Work hasn’t been started.
A platform filesystem watcher for Linux (> 2.4 or so) was toyed with but ultimately rejected out of accuracy concerns. It was far more efficient than the cross-platform implementation “warthog”, no doubt, but it lacked accuracy. Work is being done to get most of the benefits from both worlds.
There are problems with the “baseline” watcher (which I’ve named “warthog” because it’s sturdy and reliable). But those are potential efficiency losses when watcher more than a few million paths. They are, thankfully, not accuracy or safety problems.
Maybe you can see the solution emerging here?
Here’s where we’re going next:
The most efficient kernel watchers can be used on most platforms, but checked for their accuracy periodically by the “warthog” watcher.
Re: Show HN: Filesystem Watcher
#50"Watcher is extremely efficient. In most cases, even when scanning millions of paths, this library uses a near-zero amount of resources." Yea, maybe or maybe not and my first guess is maybe not. This needs at least some bullet points on HOW it does this so efficiently so that I'll keep looking. A blanket statement like this means "they hope it is efficient" or "They want it to be efficient" or "It's good in some scen…
You are right. I’ll make sure to give a deeper breakdown in the readme.