Live data from Hacker News

Show HN: Filesystem Watcher

github.com

71–80 of 80 posts

Re: Show HN: Filesystem Watcher

#71
post #29
post #16

Earlier quoted context omitted.

Wait, it doesnt hook the OS file handling routines? it actually manually rescans the filesystem?

It does one or the other. There are concerns about OS filesystem event hooks. The current solution isn’t ideal, and is being addressed here: https://github.com/e-dant/watcher/issues/10

not on windows (only platform I checked/care about)

Re: Show HN: Filesystem Watcher

#72
post #33

Looking at Win32, it scans the whole directory periodically, right? I must miss something, but how can that be called efficient?

It’s efficient because it beats kqueue while reporting events accurately. A proper benchmarking program is in the works, however manual testing does show only minimal resource usage. For more, see this issue: https://github.com/e-dant/watcher/issues/10

I only mentioned Win32. There this library is very inefficient compared to ReadDirectoryChangesW, which consumes no CPU times when nothing changes.

Re: Show HN: Filesystem Watcher

#73

When I last tried to implement this, by far the toughest part was making sure the file that’s been newly detected is done being written to. On ntfs I couldn’t find a good technique, even last modified time was not reliable. I had to watch it for changes myself.

I've done this by watching the NTFS journal which is surprisingly efficient. First I scanned the whole journal for filesystem metadata and dumped it into a SQLite database (which took about a minute), then kept it up to date which took virtually no resources. This was an absurdly faster way to search by file name, a search across the whole FS came back in milliseconds instead of Explorer's multiple minutes.

Re: Show HN: Filesystem Watcher

#74
post #49
post #38

I have personally written a similar tool and I am very curious about how this could be using a near-zero amount of resources while maintaining accuracy. As far as I know, there are two ways to implement this functionality: 1) store an in memory representation of the file system and periodically refresh the in memory state by polling the paths under watch and emitting events when differences are detected 2) hook into…

More technically, here’s what we have: A “baseline” filesystem watcher which uses only the standard library. It has been made to beat kqueue. And it does. A platform filesystem watcher for Darwin is used, but certain event properties are handled by the standard library. Namely, the event time and the path type. A platform filesystem watcher is schedule for Windows. Work hasn’t been started. A platform filesystem watc…

Have you tried using auditpipe?

Re: Show HN: Filesystem Watcher

#75
Is there a well-tested, reliable, flexible and good working tool for windows that does the same and can be installed as a service? Just watch a directory and do something that I can configure easily with a textfile?

Re: Show HN: Filesystem Watcher

#76
post #75

Is there a well-tested, reliable, flexible and good working tool for windows that does the same and can be installed as a service? Just watch a directory and do something that I can configure easily with a textfile?

Well, I am not too lazy to search but I was interested in your experience, especially with reliability.

This one looks interesting: https://github.com/emcrisostomo/fswatch

Re: Show HN: Filesystem Watcher

#78
post #67

Earlier quoted context omitted.

Never even thought of that; I don’t know. I assumed it was when a write was done. Whatever that means, I don’t know either.

Would this mean that fs event based antivirus scanners could be side-stepped by writing a payload to a file and then never closing the handler?

Good point here, I will definitely check this and report back. :)

Re: Show HN: Filesystem Watcher

#79
post #30
post #11

Does it keep working when files get overwritten by moving another over it, like some Linux text editors do?

I should test this. I haven’t seen a problem with that so far in my personal usage, so I’m inclined to say probably. That’s a good test case. I’ll make an issue.

This was a case that tripped me up when using Qt's QFileSystemWatcher. Many apps implement "atomic updates" to files by writing out to a new file (e.g. foo.txt.temp), then moving that file on top of the old one. As far as QFileSystemWatcher was concerned, the old file was deleted and its job is done (just because the new one has the same name doesn't mean it's the same file). So you have to watch the parent directory and manually implement checking for a specific file by name when the parent directory's contents changed.

Re: Show HN: Filesystem Watcher

#80
post #56

Earlier quoted context omitted.

No, its a certificate issue; probably minor, but a server-side thing to attend to by the looks of it.

There are no certificate issues in my Chrome. Are you sure your not on some MITMing VPN?

(late update for anybody reading this): looks like you were correct, (apologies to voidtools). When I click through the (local) corporate content blocker kicks in.
Post reply on HN