Live data from Hacker News

Proposal to add build graph output to GNU Make (2020)

jonasdn.blogspot.com

11–20 of 40 posts

Re: Proposal to add build graph output to GNU Make (2020)

#11
I once helped maintain a nearly full featured implementation of make in Python (pymake) that grew out of the Firefox project.

As much as I would like to support efforts like this, I feel like it is ultimately doomed to suffer from usability limitations because make does not have a static DAG. Rather, the DAG evolves dynamically as make evaluates targets. There are pesky problems like $(call) and $(eval) where you can dynamically inject make expressions into a partially evaluated DAG. And targets can generate files which are later loaded via include directives.

Dumping the make database in a machine readable format (for visualization or otherwise) would be incredibly valuable for debugging and could improve understanding. But since many make files have N>>1 "snapshots" of the internal DAG during their execution, there is a really thorny problem around when and which of these "snapshots" to use and how to stitch them together. Many make files aren't "static" enough to support exporting a single, complete, and valuable usable snapshot of their internal DAG.

If debugging is the target goal, I think a potentially better approach would be to define a function that takes an output filename and optional list of targets and writes out the point-in-time build graph when that function is called. This way makefile authors could insert probes in the make file (including at the bottom of a make file so the function is called on file load) and capture the state(s) of the build graph exactly when they need to. There could also be a specially named variable holding the names of targets that when pre- or post-evaluated would trigger the dumping of the database.

Best of luck to the person doing this work. Debugging make is notoriously hard and any progress in this area will be much welcomed by its many users.

Re: Proposal to add build graph output to GNU Make (2020)

#12

Considering that all build systems must keep a dependency graph internally, it’s surprising how many of them make it incredibly difficult to inspect why things are being rebuilt. I don’t really get it either, I’ve talked to a lot of people and getting feedback on how the build system is doing its task is one of the biggest complaints I see (others are “what is the build system doing” and “why does this have the worst…

i think one of the big problems is that when projects get unwieldly with code generation and external systems, build steps often have unintended side effects which then trigger other build rules unexpectedly.

i remember reading about some wild system on here that exposes the entire source tree via fuse so that it can enforce rules about what changes and what doesn't for each step. wish i could remember the name...

edit: ah, yes. tup. https://gittup.org/tup/

edit 2: see tup graph: https://gittup.org/tup/ex_dependencies.html

Re: Proposal to add build graph output to GNU Make (2020)

#13
Wouldn't it be possible/easier to create a new/separate script that converts the `make -p` output to the desired `.dot` graph? (Just to have this not be a thing that needs to pass through the GNU make project.)

Also I feel like on most nontrivial projects the graph would be gigantic & to get any usability you'd need something interactive where you can collapse/expand the nodes you're (not) interested in. Don't we have a bunch of JS libraries these days that provide this?

Re: Proposal to add build graph output to GNU Make (2020)

#14
Well there is makefile2graph https://github.com/lindenb/makefile2graph. It is even packaged be Debian and it does exactly that.

I use it like this from inside the Makefile:

  make-db.png:       
      make -f $(MAKEFILE) -Bnd | sed -e 's,$(MAKEDIR)/,,g' | make2graph | dot -Tpng -o $@
but with some gymnastics in order to remove very long path names (MAKEDIR) can be used from the shell directly.

Re: Proposal to add build graph output to GNU Make (2020)

#16

Considering that all build systems must keep a dependency graph internally, it’s surprising how many of them make it incredibly difficult to inspect why things are being rebuilt. I don’t really get it either, I’ve talked to a lot of people and getting feedback on how the build system is doing its task is one of the biggest complaints I see (others are “what is the build system doing” and “why does this have the worst…

I don't understand why they can't all have interactive debugging either.

Re: Proposal to add build graph output to GNU Make (2020)

#17

Ive tried this before but with a complex build the graph is unwieldy and useless

It’s probably more suited for the OP’s use case (a couple of files causing some re-building) because otherwise yeah, the build tree can be humongous. That said, isn’t there some software to visualise dot files in a more sophisticated way than just generating a picture? It would be handy to be able to collapse/expand sub graphs in a GUI, for example.

Something like this could be handy. (Disclaimer: never used it)

https://pypi.org/project/pydot/

Re: Proposal to add build graph output to GNU Make (2020)

#18
post #2

Some clarification needed, > The Android build system works by including all recipes to be built (programs / libraries / etc) using the GNU Make include directive, so that you end up with one giant Makefile that holds all rules for building the platform. Possibly to avoid the problems laid out in the paper Recursive make considered harmful. The old build system (ndk-build) works like this. Modern NDK applications are…

> Modern NDK applications are supposed to use CMake and Android's own build system is based on Soong.

Soong build files are easy to parse and you can create an Unified Dependency Graph from them. That's what I did in BGraph [0] for security applications.

[0] https://github.com/quarkslab/bgraph

Re: Proposal to add build graph output to GNU Make (2020)

#20

Considering that all build systems must keep a dependency graph internally, it’s surprising how many of them make it incredibly difficult to inspect why things are being rebuilt. I don’t really get it either, I’ve talked to a lot of people and getting feedback on how the build system is doing its task is one of the biggest complaints I see (others are “what is the build system doing” and “why does this have the worst…

Msbuild (as used in Visual Studio and dotnet core) has an amazing implementation of this: https://msbuildlog.com/

Not only can you see what was rebuilt and why, you can see all the input into the rules at all the stages of the build, conveniently hyperlinked into the master rule files provided in the SDK.

Post reply on HN