Nice! Leaving a comment to easily find this later, dont have anything to add except this looks cool
I made a real-time C/C++/Rust build visualizer
21–30 of 91 posts
Re: I made a real-time C/C++/Rust build visualizer
#22Nice, I’ve been looking for something like this for a while. I’ve noticed on my huge catkin cmake project that cmake is checking the existence of the same files hundreds of times too. Is there anything that can hook into fork() and provide a cached value after the first invocation?
My tips for speeding up builds (from making this same project but with ebpf):
- switch to ninja to avoid that exact issue since CMake + Make spawns a subprocess for every directory (use the binary from PyPi for jobserver integration)
- catkin as in ROS? rm /opt/ros/noetic/etc/catkin/profile.d/99.roslisp.sh to remove 2 python spawns per package
Re: I made a real-time C/C++/Rust build visualizer
#23> It also has 6 seconds of inactivity before starting any useful work. For comparison, ninja takes 0.4 seconds to start compiling the 2,468,083 line llvm project. Ninja is not a 100% fair comparison to other tools, because it benefits from some “baked in” build logic by the tool that created the ninja file, but I think it’s a reasonable “speed of light” performance benchmark for build systems.
This is an important observation that is often overlooked. What’s more, the changes to the information on which this “baked in” build logic is based is not tracked very precisely.
How close can we get to this “speed of light” without such “baking in”? I ran a little benchmark (not 100% accurate for various reasons but good enough as a general indication) which builds the same project (Xerces-C++) both with ninja as configured by CMake and with build2, which doesn’t require a separate step and does configuration management as part of the build (and with precise change tracking). Ninja builds this project from scratch in 3.23s while build2 builds it in 3.54s. If we omit some of the steps done by CMake (like generating config.h) by not cleaning the corresponding files, then the time goes down to 3.28s. For reference, the CMake step takes 4.83s. So a fully from-scratch CMake+ninja build actually takes 8s, which is what you would normally pay if you were using this project as a dependency.
Re: I made a real-time C/C++/Rust build visualizer
#24> It also has 6 seconds of inactivity before starting any useful work. For comparison, ninja takes 0.4 seconds to start compiling the 2,468,083 line llvm project. Ninja is not a 100% fair comparison to other tools, because it benefits from some “baked in” build logic by the tool that created the ninja file, but I think it’s a reasonable “speed of light” performance benchmark for build systems. This is an important ob…
> What’s more, the changes to the information on which this “baked in” build logic is based is not tracked very precisely.
kbuild handles this on top of Make by having each target depend on a dummy file that gets updated when e.g. the CFLAGS change. It also treats Make a lot more like Ninja (e.g. avoiding putting the entire build graph into every Make process) -- I'd be interested to see how it compares.
Re: I made a real-time C/C++/Rust build visualizer
#25This is great. I was skeptical from the title but the implementation is very clever. This could be a super super useful tool for the industry.
Re: I made a real-time C/C++/Rust build visualizer
#26That's interesting.
Re: I made a real-time C/C++/Rust build visualizer
#27This seems like a good place to integrate a Bazel Build Event Protocol stream consumer.
Re: I made a real-time C/C++/Rust build visualizer
#28This seems like a good place to integrate a Bazel Build Event Protocol stream consumer.
I was going to comment that "what the fork" might not work for a client/server build system like Bazel, but now I have something to google instead.
Re: I made a real-time C/C++/Rust build visualizer
#29I love the visualization, I think it's great information and will be very helpful to whoever uses it.
I would think about a different name. Often names are either meant to be funny or just unique nonsense but something short and elegantly descriptive (like BuildViz etc.) can go a long way to making it seem more legitimate and being more widely used.
Re: I made a real-time C/C++/Rust build visualizer
#30I am extremely interested in this. I am stuck in an environment with CMake, GCC and Unix Make (no clang, no ninja) and getting detailed information about WHY the build is taking so long is nearly impossible. It's also a bit of a messy build with steps like copying a bunch of files from the source into the build folder. Multiple languages (C, C++, Fortran, Python), custom cmake steps, etc. If this tool can handle that…
I wrote a little GCC plugin for compile time tracing/profiling, if that's something you're interested in: https://github.com/royjacobson/externis