Tracking Changes in Directories with Python • Hacker News Board

Re: Tracking Changes in Directories with Python

#21

Post by blueblob » Fri, Dec 13, 2013, 3:03 PM UTC

tsileo wrote: post #11

aray wrote: post #9

Earlier quoted context omitted.

I'm not sure what you mean by continually running, but inotifywait is basically just waiting on an event. As long as the process sticks around it doesn't have to do anything until it gets an inotify.

I don't want to have a running process just for this, I want to be able to take a "snapshot"/"index", store some data on Glacier, and the index on S3, and later, given the index, be able to compute deltas without accessing the full archive store on Glacier.

Is there a reason that you want it in python? Couldn't you do this with diff -q if you only want to know what data has changed? and then use diff to get the deltas?

Re: Tracking Changes in Directories with Python

#22

Post by johtso » Fri, Dec 13, 2013, 3:38 PM UTC

The watchdog library is great for this. It comes with an API and a command line tool: https://github.com/gorakhargosh/watchdog

Compatibility:

  Linux 2.6 (inotify)
  Mac OS X (FSEvents, kqueue)
  FreeBSD/BSD (kqueue)
  Windows (ReadDirectoryChangesW with I/O completion ports; ReadDirectoryChangesW worker threads)
  OS-independent (polling the disk for directory snapshots and comparing them periodically; slow and not recommended)

Re: Tracking Changes in Directories with Python

#23

Post by rytis » Wed, Dec 18, 2013, 7:37 AM UTC

nodata wrote: post #19

rytis wrote: post #15

Earlier quoted context omitted.

Or even use incrond [0], and get it call your script that does whatever needs to be done with the modified files. [0] http://linux.die.net/man/8/incrond EDIT: As a use case example I use this to detect when a new cert request is made to my Puppet master server. Once a CSR file is created, I check if the host is created by our provisioning system, and if so, then sign it. EDIT2: This will eventually go into 3.4.0 - ht…

How does the puppet master get the csr unless the host has been provisioned?

The host is provisioned using some other automation tool, then when it comes up, it runs puppet agent. Puppet agent makes cert sign request (at which point csr file is created on puppet server).

Puppet server doesn't have this host in autosign.conf, so doesn't automatically sign the cert. And this is where incron kicks in and runs my script. The script queries the provisioning tool, sees that the host is in there, and runs puppet cert sign command.