Tracking Changes in Directories with Python
thomassileo.com
Tracking Changes in Directories with Python
1–10 of 23 posts
Re: Tracking Changes in Directories with Python
#2If you do not want to pay for the overhead, I recommend inotify-tools. They have Python bindings too: https://github.com/seb-m/pyinotify
Re: Tracking Changes in Directories with Python
#3Since he asks for feedback at the end...
Using sha256 just to compute changes is probably overkill. Using md5 instead is almost certainly adequate and will be a good deal faster.
Re: Tracking Changes in Directories with Python
#4Since he asks for feedback at the end... Using sha256 just to compute changes is probably overkill. Using md5 instead is almost certainly adequate and will be a good deal faster.
> good deal faster
You could always try it and see :)
Example test: openssl speed md5 sha256
As for why it was chosen, I think it's because there are known examples of MD5 hash collisions (though the likelihood of it on a filesystem is remote) and likely SHA-1 was skipped because it's considered 'likely' a collision could be created (though so far only with weakened versions of SHA-1).
But - all this to say: The chances of having two files with the same MD5 hash that are identical in size is vanishingly small. As such, for the known MD5 collision mechanisms, the different file size would be enough evidence something has changed.
... Why he didn't include file size in the metadata check, I can't tell you. Timestamps can be faked - but generating a hash collision with a file of equal size is a Hard problem.
Re: Tracking Changes in Directories with Python
#5I can't tell exactly what the goal is, but inotify might also be a simpler solution.
(Specifically, if the goal is to monitor changes as they happen and the service can be assumed to be continually running.)
Re: Tracking Changes in Directories with Python
#6I can't tell exactly what the goal is, but inotify might also be a simpler solution. (Specifically, if the goal is to monitor changes as they happen and the service can be assumed to be continually running.)
And the command line inotifywait - which I've used from Python to monitor changes to directories. There is even a Windows port:
Re: Tracking Changes in Directories with Python
#7I can't tell exactly what the goal is, but inotify might also be a simpler solution. (Specifically, if the goal is to monitor changes as they happen and the service can be assumed to be continually running.)
The service is not continually running, I use this method to make incremental backups with archives stored on AWS Glacier and meta-data stored on S3 (the index is stored on S3, and I can't access files on Glacier to compute deltas).
Re: Tracking Changes in Directories with Python
#8I can't tell exactly what the goal is, but inotify might also be a simpler solution. (Specifically, if the goal is to monitor changes as they happen and the service can be assumed to be continually running.)
And the command line inotifywait - which I've used from Python to monitor changes to directories. There is even a Windows port: https://github.com/thekid/inotify-win
pyinotify is also pretty simple to use: https://github.com/seb-m/pyinotify
Re: Tracking Changes in Directories with Python
#9I can't tell exactly what the goal is, but inotify might also be a simpler solution. (Specifically, if the goal is to monitor changes as they happen and the service can be assumed to be continually running.)
The service is not continually running, I use this method to make incremental backups with archives stored on AWS Glacier and meta-data stored on S3 (the index is stored on S3, and I can't access files on Glacier to compute deltas).
I'm not sure what you mean by continually running, but inotifywait is basically just waiting on an event. As long as the process sticks around it doesn't have to do anything until it gets an inotify.
Re: Tracking Changes in Directories with Python
#10Since he asks for feedback at the end... Using sha256 just to compute changes is probably overkill. Using md5 instead is almost certainly adequate and will be a good deal faster.
> good deal faster You could always try it and see :) Example test: openssl speed md5 sha256 As for why it was chosen, I think it's because there are known examples of MD5 hash collisions (though the likelihood of it on a filesystem is remote) and likely SHA-1 was skipped because it's considered 'likely' a collision could be created (though so far only with weakened versions of SHA-1). But - all this to say: The chan…
Thanks for the feedback!
I haven't thought about the filesize/hash to reduce collision, but I chose to stick with the last-modified time in the article, because it can takes hours computing hashes for a big directory tree.
Tools like rsync relies on last-modified time by default, and since I want to use this to track my own files, I won't fake it, so I think it's not a big deal?