Watchman: Faster builds with large source trees

This is cross-platform, which inotify isn't. Also, isn't inotify pretty low-level? It looks like FB has consolidated common inotify helper code into a reusable daemon.

Re: Watchman: Faster builds with large source trees

#6

Post by yid » Thu, May 30, 2013, 9:44 PM UTC

_mpu wrote: post #4

yid wrote: post #2

Not sure how this is different from inotify+md5+make?

Not sure how inotify is different from while+stat?

Your snark is unwarranted. While+stat in a loop is a poll loop in userspace, inotify is a push mechanism in kernel space.

My point was that the additional functionality of this significantly-sized package beyond running inotify+md5+make in a shell script was unclear.

Re: Watchman: Faster builds with large source trees

#7

Post by kevingadd » Thu, May 30, 2013, 9:57 PM UTC

thelarry wrote: post #3

Compiling while editing. Intredasting.... I assume there is some crazy dependency algorithm.

Traditional build systems are all dependency algorithms. The interesting thing here is that they run a service to track file modifications so it's not necessary to stat() the entire hard drive to find out what has changed, so you can do incremental rebuilds very cheaply.

Re: Watchman: Faster builds with large source trees

#8

Post by norswap » Thu, May 30, 2013, 10:00 PM UTC

What is the difference with guard (https://github.com/guard/guard), except the "service" aspect?

Re: Watchman: Faster builds with large source trees

#9

Post by evmar » Thu, May 30, 2013, 10:13 PM UTC

If I'm reading that graph right, the red line is at 10kmsec, i.e. 10 seconds?

Incremental build times are near and dear to my heart; I spent a lot of time making the Chrome incremental build fast, resulting in this tool: http://martine.github.io/ninja/ . In developing Ninja I was surprised to discover that Linux stat() with a warm disk cache is very fast -- well under 100ms to stat the ~40k source files Chrome uses in its build (see the "node stat" lines here: https://github.com/martine/ninja/wiki/Timing-Numbers ). At its best point I think we got the one-file-changed build/compile/link cycle of Chrome (a ~70mb C++ binary) to around 5 seconds.

Of course, Facebook's problem is surely very different -- their scale could be many more files, and perhaps the programs their engineers run while developing cause their disk caches to flush more frequently. Just found it interesting to worry about the cost of stats.

Re: Watchman: Faster builds with large source trees

#10

Post by Terretta » Thu, May 30, 2013, 10:23 PM UTC

_mpu wrote: post #4

yid wrote: post #2

Not sure how this is different from inotify+md5+make?

Not sure how inotify is different from while+stat?

Trivial example use of inotify (or rather, inotifywait) to restart Apache after you edit the Apache conf file or add/edit/remove a site config:

    while inotifywait -e attrib,modify /etc/httpd/conf/httpd.conf -e attrib,modify,create,delete,move -r /etc/httpd/sites-enabled ; do
      /sbin/service httpd graceful && echo "`date -u --rfc-3339=seconds` httpd graceful" >> /etc/httpd/conf/httpd-conf.log
    done

Shows how to monitor a single file and a directory, and do a sequence of commands if events happen. Unlike while stat, this doesn't spam checking the file system for changes, it waits until the kernel notifies that a change happened.

To "daemonize" this, tack on an invocation test, perhaps:

    #!/bin/bash
    if [ "x$1" != "x--" ]; then
    $0 -- 1> /etc/httpd/conf/watchconf.log 2> /etc/httpd/conf/watchconf-err.log &
    exit 0
    fi
    
    while inotifywait -e attrib,modify /etc/httpd/conf/httpd.conf -e attrib,modify,create,delete,move -r /etc/httpd/sites-enabled ; do
      /sbin/service httpd graceful && echo "`date -u --rfc-3339=seconds` httpd graceful" >> /etc/httpd/conf/httpd-conf.log
    done