The index page makes it sound like a parody of something, took me a while to figure it is functional software.
Tup – A file-based build system for Linux, OS X, and Windows
31–40 of 110 posts
Re: Tup – A file-based build system for Linux, OS X, and Windows
#32Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! On a related note, I've always wondered if it was possible to have a build system based on dynamic library injection / strace. The idea would be that you just write your build rules in shell script. Then, you run it with a special shell that catches open(), etc. in child processes (via library injection, etc). These syste…
See fabricate.py: https://github.com/SimonAlfie/fabricate
Here's an earlier comment I wrote about it, which is also in the middle of a discussion about Tup, funnily enough: https://news.ycombinator.com/item?id=4190804
I'm still using Fabricate quite happily today for personal projects in a variety of languages, though I've come to realise that the main downside of it is that programs run slower under strace - a typical gcc invocation to create a single object file might take say, 40% longer. Fabricate.py is also then quite slow to parse the textual output from strace.
Inspired by https://blog.nelhage.com/2010/08/write-yourself-an-strace-in... I wrote a mini-strace that's hard coded to catch the open() and other relevant calls as fast as possible. That helped a fair bit. I've done a lot of hacking on my own copy of Fabricate too which helped a bit, but ultimately I ought to make it do the dependency parsing and saving in a seperate thread, which should help a lot - I'm still not doing parallel builds so I do have spare cores sitting there doing nothing.
I tried a FUSE backend (https://github.com/SimonAlfie/fabricate/pull/60) whereby the filesystem tells you what's being accessed, and that's practically the same as full speed but the downside is that the path jiggery-pokery involved then causes some subsequent problems with eg. gdb not being able to find source files. I've gone back to the strace method for now.
Despite all these niggles, I still use this system wherever possible because it's so, so good having a 'proper' programming language to run my builds, and being able to work forwards in the manner of a shell script. And yeah the automatic detection of changes ultimately saves a lot of time. I can share some of my hacked-up scripts if anyone's interested.
Re: Tup – A file-based build system for Linux, OS X, and Windows
#33Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! On a related note, I've always wondered if it was possible to have a build system based on dynamic library injection / strace. The idea would be that you just write your build rules in shell script. Then, you run it with a special shell that catches open(), etc. in child processes (via library injection, etc). These syste…
There is such a build system, but I can't remember the name right now. It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic). The downside is that it's Linux only. If anyone remembers the name, please do share.
https://github.com/rizsotto/Bear/blob/master/README.md
Bear do something similar.
Re: Tup – A file-based build system for Linux, OS X, and Windows
#34Earlier quoted context omitted.
> Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! To be honest, the only time I haven't felt like an army of build rules isn't a weight I have to carry uphill is when I use Lein (which is more of a build project tool as opposed to something as elemental as tup). Systems like SBT and Ant end up with things that feel to me like a complete and total reliance on hand-toole…
The one thing I like about Gulp is that it makes it easy to make a project out of your build process, to basically enforce your own convention over configuration. I have a single, canonical build process for all of my projects and it gets installed through NPM like every other dependency. The individual gulpfiles for my projects end up being very simple and readable. It has the side effect of drastically cleaning up…
Recently I had a local Gulp expert who had written a plugin try and help me with a simple problem:
I produce a .json of load balancers from a CF stack. We need to run a command on every entry in the map to annotate it with extra information from another AWS call. It's a tricky problem not amenable to usual shell scripting (unless you use a shell with a native notion of maps and json).
For the life of us, we could not find a way to make Gulp do this without essentially writing a solid block of Javascript that just does exactly what we wanted (in which case, why use anything but the shell script approach?).
I understand that stream rejoining is a complex problem, but Gulp seems ideologically opposed to a more generic approach here. They get mad when people try to inject non-file based values into the system, and actively close off alternatives and shame them on their github issues.
Gulp is a complete mess, in my opinion.
Re: Tup – A file-based build system for Linux, OS X, and Windows
#35I'd not heard of tup, thought I'd try it out on Windows. Unfortunately I hit a bug straight away: Tup is not directly compatible with MSVC 2015 (without disabling VCToolsTelemetry.dat generation in the registry) [0].
I don't fancy adopting a tool that forces me to opt-out of being able to send compiler debug telemetry to Microsoft the next time I hit a compiler bug.
There is a nice (but a bit dated, 2010) review here [1] which discusses some other features and shortcomings.
[0] https://github.com/gittup/tup/issues/182
[1] 2010: https://chadaustin.me/2010/06/scalable-build-systems-an-anal...
Re: Tup – A file-based build system for Linux, OS X, and Windows
#36Previous HN discussion from Nov 1, 2014:
Re: Tup – A file-based build system for Linux, OS X, and Windows
#37Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! On a related note, I've always wondered if it was possible to have a build system based on dynamic library injection / strace. The idea would be that you just write your build rules in shell script. Then, you run it with a special shell that catches open(), etc. in child processes (via library injection, etc). These syste…
Tup does this.
See here: http://gittup.org/tup/ex_a_first_tupfile.html
> The trick is that tup instruments all commands that it executes in order to determine what files were actually read from (the inputs) and written to (the outputs). When the C preprocessor opens the header file, tup will notice that and automatically add the dependency. In fact, we don't have to specify the C input file either, but you can leave that in there for now since we'll use it in the next section.
Re: Tup – A file-based build system for Linux, OS X, and Windows
#38I'd not heard of tup, thought I'd try it out on Windows. Unfortunately I hit a bug straight away: Tup is not directly compatible with MSVC 2015 (without disabling VCToolsTelemetry.dat generation in the registry) [0]. I don't fancy adopting a tool that forces me to opt-out of being able to send compiler debug telemetry to Microsoft the next time I hit a compiler bug. There is a nice (but a bit dated, 2010) review here…
[deleted]
Re: Tup – A file-based build system for Linux, OS X, and Windows
#39Earlier quoted context omitted.
> That said it still beats the crap out GNU Make How? Their description was useless in describing why it's "obviously" so much better.
GNU make starts of with targets, and then talks the build graph (target -> source) to find source files which will re-build the targets. And GNU Make has to scan the files. Tup starts off with the list of sources which have changed, and then walks the build graph (source -> target) to find out which targets need to be rebuilt. And Tup gets notified of file changes. So where Make scans all targets and all source files…
GNU Make doesn't take exponential time, it takes ~linear time. The Y value on the graphs is exponential, because the input is growing exponentially.
The graph is just rubbing in that a noop Tup build is O(1) instead of Make's O(n) by showing crazy large n. Which is certainly great on Tup's part![1]
The Tup author's description of Tup's algorithm in comparison has always rubbed me the wrong way, though. It isn't some entirely new generation of build system like they present in the Tup paper. It's just caching stat(2) between iterations[2]. The build naturally flows down like in the Make diagram on the main Tup page; but with Tup we cache the status of each node, and become informed when one of them becomes invalid; then the upward flowing arrows of the Tup diagram is the cache-invalidation of the lower nodes flowing up to the higher nodes. This is the totally natural way of implementing caching if we were to add that to GNU Make.
So why don't GNU Make and other systems cache the DAG of stat(2) calls? As the saying says, cache invalidation is hard. Tup gets away with caching by receiving an a priori list of filesystem changes, so it knows which nodes to invalidate; other build systems have to scan the filesystem for changes. Tup gets this list of changes by having a FUSE filesystem sit between the user and the actual filesystem, logging all changes. While this is great for many users, it isn't quite general. It will break if we edit anything "offline", perhaps on a thumb drive on another computer. It will break in a number of different NFS setups. I don't mean to say that the caching doesn't have value, but it isn't the silver bullet that the Tup author makes it out to be.
[1]: In the Tup paper, the author gives some fancy O(...) expressions. These are... odd; they're kind of hand-wavey, as they are based on expected fan-out of directories, and don't really give a good apples-to-apples comparison. They certainly are valuable to analyze, but are pretty misleading if you aren't critical.
[2]: OK, that's not strictly true. The "Beta build system" (the next generation of build systems that Tup is supposedly part of) algorithm also has the feature that it will automatically remove a generated file that was part of the DAG, but isn't anymore. This is a nifty feature that is made possible by the caching; but IMO it doesn't constitute a core change.
Re: Tup – A file-based build system for Linux, OS X, and Windows
#40Tup's main problem is it's unusual, and it doesn't have a library of build rules. But it's fast! On a related note, I've always wondered if it was possible to have a build system based on dynamic library injection / strace. The idea would be that you just write your build rules in shell script. Then, you run it with a special shell that catches open(), etc. in child processes (via library injection, etc). These syste…
There is such a build system, but I can't remember the name right now. It tracks system calls to see every file opened by the compiler to produce exact dependency graphs (assuming compiler is deterministic). The downside is that it's Linux only. If anyone remembers the name, please do share.
I wrote something called auto-buildrequires which lets you figure out build dependencies automatically, and works roughly as you describe. https://people.redhat.com/~rjones/auto-buildrequires/