Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking Changes in Directories with Python

thomassileo.com

Post reply on HN
Print view View original

11–20 of 23 posts

Re: Tracking Changes in Directories with Python

#11

Post by tsileo »

aray wrote: post #9
tsileo wrote: post #7

Earlier quoted context omitted.

The service is not continually running, I use this method to make incremental backups with archives stored on AWS Glacier and meta-data stored on S3 (the index is stored on S3, and I can't access files on Glacier to compute deltas).

I'm not sure what you mean by continually running, but inotifywait is basically just waiting on an event. As long as the process sticks around it doesn't have to do anything until it gets an inotify.

I don't want to have a running process just for this, I want to be able to take a "snapshot"/"index", store some data on Glacier, and the index on S3, and later, given the index, be able to compute deltas without accessing the full archive store on Glacier.

Re: Tracking Changes in Directories with Python

#12

Post by arethuza »

icebraining wrote: post #8
arethuza wrote: post #6

Earlier quoted context omitted.

And the command line inotifywait - which I've used from Python to monitor changes to directories. There is even a Windows port: https://github.com/thekid/inotify-win

pyinotify is also pretty simple to use: https://github.com/seb-m/pyinotify

Thanks, I was after something that would work from Python on Linux and Windows and calling out to inotifywait seemed to work well for that - this is for my home grown RESTful Dropbox-lite application which is sitting at 95% complete...

Re: Tracking Changes in Directories with Python

#13

Post by netnichols »

tsileo wrote: post #10
pudquick wrote: post #4

Earlier quoted context omitted.

> good deal faster You could always try it and see :) Example test: openssl speed md5 sha256 As for why it was chosen, I think it's because there are known examples of MD5 hash collisions (though the likelihood of it on a filesystem is remote) and likely SHA-1 was skipped because it's considered 'likely' a collision could be created (though so far only with weakened versions of SHA-1). But - all this to say: The chan…

Thanks for the feedback! I haven't thought about the filesize/hash to reduce collision, but I chose to stick with the last-modified time in the article, because it can takes hours computing hashes for a big directory tree. Tools like rsync relies on last-modified time by default, and since I want to use this to track my own files, I won't fake it, so I think it's not a big deal?

It's not just that it could be faked, it could be an accident that you modify a file but the file modification date is not changed. For example, say you edit a photo, but later you run a script that sets the file's modification date to the EXIF data in the photo.

So I guess the point is that also including the file size will be one more (fast) data point to help ensure 'accurate' change tracking, without adding the overhead of computing content hashes.

Re: Tracking Changes in Directories with Python

#15

Post by rytis »

yummyfajitas wrote: post #5

I can't tell exactly what the goal is, but inotify might also be a simpler solution. (Specifically, if the goal is to monitor changes as they happen and the service can be assumed to be continually running.)

Or even use incrond [0], and get it call your script that does whatever needs to be done with the modified files.

[0] http://linux.die.net/man/8/incrond

EDIT: As a use case example I use this to detect when a new cert request is made to my Puppet master server. Once a CSR file is created, I check if the host is created by our provisioning system, and if so, then sign it.

EDIT2: This will eventually go into 3.4.0 - https://projects.puppetlabs.com/issues/7244

Re: Tracking Changes in Directories with Python

#16

Post by cmsd2 »

tsileo wrote: post #7
yummyfajitas wrote: post #5

I can't tell exactly what the goal is, but inotify might also be a simpler solution. (Specifically, if the goal is to monitor changes as they happen and the service can be assumed to be continually running.)

The service is not continually running, I use this method to make incremental backups with archives stored on AWS Glacier and meta-data stored on S3 (the index is stored on S3, and I can't access files on Glacier to compute deltas).

I've been thinking about the best way to do this, and I too didn't want to rely on having a backup script running all the time.

Windows seems to have a low-level api for querying the filesystem/vfs for changes since you last looked: http://msdn.microsoft.com/en-us/library/windows/desktop/aa36...

And BTRFS has some ability to do this with find-new: http://www.tummy.com/blogs/2010/11/01/fun-with-btrfs-what-fi...

It's nice that btrfs has these new interesting features, also see send/receive, but it's not available in the vfs, and I suppose never will be.

Re: Tracking Changes in Directories with Python

#17

Post by tsileo »

netnichols wrote: post #13
tsileo wrote: post #10

Earlier quoted context omitted.

Thanks for the feedback! I haven't thought about the filesize/hash to reduce collision, but I chose to stick with the last-modified time in the article, because it can takes hours computing hashes for a big directory tree. Tools like rsync relies on last-modified time by default, and since I want to use this to track my own files, I won't fake it, so I think it's not a big deal?

It's not just that it could be faked, it could be an accident that you modify a file but the file modification date is not changed. For example, say you edit a photo, but later you run a script that sets the file's modification date to the EXIF data in the photo. So I guess the point is that also including the file size will be one more (fast) data point to help ensure 'accurate' change tracking, without adding the o…

I think I will add the file size to the index, since it's really cheap.

Thanks!

Re: Tracking Changes in Directories with Python

#18

Post by xradionut »

cmsd2 wrote: post #16
tsileo wrote: post #7

Earlier quoted context omitted.

The service is not continually running, I use this method to make incremental backups with archives stored on AWS Glacier and meta-data stored on S3 (the index is stored on S3, and I can't access files on Glacier to compute deltas).

I've been thinking about the best way to do this, and I too didn't want to rely on having a backup script running all the time. Windows seems to have a low-level api for querying the filesystem/vfs for changes since you last looked: http://msdn.microsoft.com/en-us/library/windows/desktop/aa36... And BTRFS has some ability to do this with find-new: http://www.tummy.com/blogs/2010/11/01/fun-with-btrfs-what-fi... It's n…

The Windows solution would seem to only work if you have admin access to the specific server or the rights to access the journal.

Re: Tracking Changes in Directories with Python

#19

Post by nodata »

rytis wrote: post #15
yummyfajitas wrote: post #5

I can't tell exactly what the goal is, but inotify might also be a simpler solution. (Specifically, if the goal is to monitor changes as they happen and the service can be assumed to be continually running.)

Or even use incrond [0], and get it call your script that does whatever needs to be done with the modified files. [0] http://linux.die.net/man/8/incrond EDIT: As a use case example I use this to detect when a new cert request is made to my Puppet master server. Once a CSR file is created, I check if the host is created by our provisioning system, and if so, then sign it. EDIT2: This will eventually go into 3.4.0 - ht…

How does the puppet master get the csr unless the host has been provisioned?

Re: Tracking Changes in Directories with Python

#20

Post by dded »

pudquick wrote: post #4
netnichols wrote: post #3

Since he asks for feedback at the end... Using sha256 just to compute changes is probably overkill. Using md5 instead is almost certainly adequate and will be a good deal faster.

> good deal faster You could always try it and see :) Example test: openssl speed md5 sha256 As for why it was chosen, I think it's because there are known examples of MD5 hash collisions (though the likelihood of it on a filesystem is remote) and likely SHA-1 was skipped because it's considered 'likely' a collision could be created (though so far only with weakened versions of SHA-1). But - all this to say: The chan…

>> good deal faster

>> You could always try it and see :)

>> Example test: openssl speed md5 sha256

When I was looking for a fast hash easily called from Python, I settled on adler32 (as the fastest) after some trial-and-error on my files. I don't now recall all the utilities/functions that I tried, but they certainly included md5, sha1, and crc32. I only needed to test for accidental corruption, and only computed the hashes if meta data matched.

Post reply on HN