Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I made a real-time C/C++/Rust build visualizer

danielchasehooper.com

Post reply on HN
Print view View original

81–90 of 91 posts

Re: I made a real-time C/C++/Rust build visualizer

#81

Post by Night_Thastus »

wakawaka28 wrote: post #50
Night_Thastus wrote: post #10

I am extremely interested in this. I am stuck in an environment with CMake, GCC and Unix Make (no clang, no ninja) and getting detailed information about WHY the build is taking so long is nearly impossible. It's also a bit of a messy build with steps like copying a bunch of files from the source into the build folder. Multiple languages (C, C++, Fortran, Python), custom cmake steps, etc. If this tool can handle that…

It's not "nearly impossible" but actually built in: https://cmake.org/cmake/help/latest/manual/cmake.1.html#cmdo... For the actual compile time you can easily insert a wrapper script. To be honest I haven't done that in over 4 years, but it has been done by many and it is easy. There may be times when CMake itself is the bottleneck but it is almost certainly an issue with your dependencies and so on. CMake has many f…

Does that just profile cmake's time to configure and generate, or the underlying compile of each file as well? Configuration and generation are only seconds when done from scratch - the build is more like 20 minutes.

Just trying to add that argument with 3.26.5 on Rocky Linux 9, I get 'Unknown argument --profiling-format=google-trace'.

Not sure why, as cmake --help clearly states it should be there...

  --profiling-format=     = Output data for profiling CMake scripts.
                                 Supported formats: google-trace
  --profiling-output=    = Select an output path for the profiling data
                                 enabled through --profiling-format.

Re: I made a real-time C/C++/Rust build visualizer

#82

Post by tom_ »

Night_Thastus wrote: post #81
wakawaka28 wrote: post #50

Earlier quoted context omitted.

It's not "nearly impossible" but actually built in: https://cmake.org/cmake/help/latest/manual/cmake.1.html#cmdo... For the actual compile time you can easily insert a wrapper script. To be honest I haven't done that in over 4 years, but it has been done by many and it is easy. There may be times when CMake itself is the bottleneck but it is almost certainly an issue with your dependencies and so on. CMake has many f…

Does that just profile cmake's time to configure and generate, or the underlying compile of each file as well? Configuration and generation are only seconds when done from scratch - the build is more like 20 minutes. Just trying to add that argument with 3.26.5 on Rocky Linux 9, I get 'Unknown argument --profiling-format=google-trace'. Not sure why, as cmake --help clearly states it should be there... --profiling-for…

You might need to specify --profiling-output= as well. I get an error fdrom cmake 3.31 if only the format is provided: CMake Error: --profiling-format specified but no --profiling-output!

Anyway, it looks like it only profiles the configure/generate steps. Not much use on Linux, but on Windows/macOS, perhaps. Due to lack of any standard package manager, it's a good idea to build every dependency from source on those OSs, and the time can mounts up.

My project is not that large, but it takes 1 minute to configure from scratch on Windows, and 10 minutes (!) on macOS.

Re: I made a real-time C/C++/Rust build visualizer

#83

Post by Night_Thastus »

tom_ wrote: post #82
Night_Thastus wrote: post #81

Earlier quoted context omitted.

Does that just profile cmake's time to configure and generate, or the underlying compile of each file as well? Configuration and generation are only seconds when done from scratch - the build is more like 20 minutes. Just trying to add that argument with 3.26.5 on Rocky Linux 9, I get 'Unknown argument --profiling-format=google-trace'. Not sure why, as cmake --help clearly states it should be there... --profiling-for…

You might need to specify --profiling-output= as well. I get an error fdrom cmake 3.31 if only the format is provided: CMake Error: --profiling-format specified but no --profiling-output! Anyway, it looks like it only profiles the configure/generate steps. Not much use on Linux, but on Windows/macOS, perhaps. Due to lack of any standard package manager, it's a good idea to build every dependency from source on those…

I specified both. In any case, yeah, cmake time isn't useful in my case.

Bizarre that you see 10 minutes on MacOS. Something's definitely busted there. It's not even that bad for me on Windows, and that's saying something.

Re: I made a real-time C/C++/Rust build visualizer

#84

Post by Night_Thastus »

unddoch wrote: post #30
Night_Thastus wrote: post #10

I am extremely interested in this. I am stuck in an environment with CMake, GCC and Unix Make (no clang, no ninja) and getting detailed information about WHY the build is taking so long is nearly impossible. It's also a bit of a messy build with steps like copying a bunch of files from the source into the build folder. Multiple languages (C, C++, Fortran, Python), custom cmake steps, etc. If this tool can handle that…

I wrote a little GCC plugin for compile time tracing/profiling, if that's something you're interested in: https://github.com/royjacobson/externis

I figured out a way to set it at a top level, so it only happens with C++ files:

target_compile_options(${NAME} PUBLIC

$:-fplugin=externis -fplugin-arg-externis-trace-dir=(where I want to put traces)>

)

But as I suspected, it is not a single trace file. It's thousands of trace files. Is there some way to collate all the data into one larger picture of how the build progressed?

Re: I made a real-time C/C++/Rust build visualizer

#85

Post by tom_ »

Night_Thastus wrote: post #83
tom_ wrote: post #82

Earlier quoted context omitted.

You might need to specify --profiling-output= as well. I get an error fdrom cmake 3.31 if only the format is provided: CMake Error: --profiling-format specified but no --profiling-output! Anyway, it looks like it only profiles the configure/generate steps. Not much use on Linux, but on Windows/macOS, perhaps. Due to lack of any standard package manager, it's a good idea to build every dependency from source on those…

I specified both. In any case, yeah, cmake time isn't useful in my case. Bizarre that you see 10 minutes on MacOS. Something's definitely busted there. It's not even that bad for me on Windows, and that's saying something.

For the record: the cmake profiling data did help a bit. On macOS it seems that the configure stage does 200+ try_compiles, largely due to SDL2, and each one somehow takes ~2.3 seconds. So that's 460 seconds right there. And regarding the total time, it's actually more like 480 seconds. (I must have misremembered! Or perhaps my laptop is measurably faster when its integrated GPU isn't driving 2 external displays.)

On Windows: 50 try_compiles, about half and half SDL2 and libuv, and each one takes ~1 second.

I don't think either of these try_compile turnaround times is acceptable (what is it doing?! I bet it's like 50 ms on Linux) but the total figure does now feel a bit less mysterious.

Re: I made a real-time C/C++/Rust build visualizer

#86

Post by zorgmonkey »

1718627440 wrote: post #79
dhooper wrote: post #77

Earlier quoted context omitted.

I used the term "early access" when I should've used "private beta". The sign up is just for me to have an email addresses to send the private beta when the next update goes out, and I'll follow up for feedback. Nothing sinister is happening.

> to send the private beta when the next update goes out Meaning I only can try the software after sign up, or did I miss the obvious repo link? What language is this project written in, what build system does it use itself? I just don't feel comfortable pasting my email address into a Google site. I can use a temp mail, but I will loose access to it in a few minutes/hours, so I don't know if that would annoy you. Wh…

You can make an email you don't care about with protonmail, I recommend them cause they don't require you to enter an existing email address or a phone number when signing up.

Re: I made a real-time C/C++/Rust build visualizer

#88

Post by entelechy »

flakes wrote: post #61
entelechy wrote: post #31

Love it! We did something similar using strace/dtruss back in 2018 with https://buildinfer.loopperfect.com/ and were generating graphs (using eg. graphviz and perfetto.dev) and BUCK files on the back of that Whilst we regrettably never came around to package it as a propper product, we found it immensly valuable in our consulting work, to pinpoint issues and aid the conversion to BUCK/Bazel. We used graphviz, https:/…

Curious, what were you using for doing syscall logging? LD_PRELOAD tricks, or ebpf filtering?

mostly strace and it's macos equivalent; Later we moved to ptrace and ebpf. LD_PRELOAD unfortunately doesn't work for statically linked libc. There are also kernel probes but didn't like that it required root permissions...

Re: I made a real-time C/C++/Rust build visualizer

#89

Post by touisteur »

pklausler wrote: post #35
Night_Thastus wrote: post #10

I am extremely interested in this. I am stuck in an environment with CMake, GCC and Unix Make (no clang, no ninja) and getting detailed information about WHY the build is taking so long is nearly impossible. It's also a bit of a messy build with steps like copying a bunch of files from the source into the build folder. Multiple languages (C, C++, Fortran, Python), custom cmake steps, etc. If this tool can handle that…

strace might help, if you have it.

perf record -e "syscalls:,sched:" is my go-to there - or just apply it on whole-system, aasone does.

Re: I made a real-time C/C++/Rust build visualizer

#90

Post by wakawaka28 »

Night_Thastus wrote: post #81
wakawaka28 wrote: post #50

Earlier quoted context omitted.

It's not "nearly impossible" but actually built in: https://cmake.org/cmake/help/latest/manual/cmake.1.html#cmdo... For the actual compile time you can easily insert a wrapper script. To be honest I haven't done that in over 4 years, but it has been done by many and it is easy. There may be times when CMake itself is the bottleneck but it is almost certainly an issue with your dependencies and so on. CMake has many f…

Does that just profile cmake's time to configure and generate, or the underlying compile of each file as well? Configuration and generation are only seconds when done from scratch - the build is more like 20 minutes. Just trying to add that argument with 3.26.5 on Rocky Linux 9, I get 'Unknown argument --profiling-format=google-trace'. Not sure why, as cmake --help clearly states it should be there... --profiling-for…

That does just do CMake time. Sorry if my comment was confusing. I forgot that I used a wrapper or something myself to collect the compile times. Anyway, my point is, you aren't the first person to have this issue. I've definitely profiled compile times before and viewed a flame graph with some gadget made by Google. The easiest way you to do it may depend on your compiler and build system. You could attack the problem from Ninja or Make, or by substituting your compiler binary for a wrapper script.

I also want to point out that if you want Ninja, it is a very minimal binary to build with no dependencies that wouldn't already be installed.

This is one of the simpler ways to find out what's going on: https://alangrow.com/blog/profiling-every-command-in-a-makef... Instead of replacing the shell you can also replace CC or CXX variables (if I recall properly) either in CMake or the Makefile to replace your compiler with a wrapper script that logs stuff.

More generalized Makefile profiling: https://github.com/konturio/make-profiler

Profiling with Clang: https://aras-p.info/blog/2019/01/16/time-trace-timeline-flam... (Maybe what I used last time...)

I am pretty sure you can get shiny outputs from a free profiling tool, but I think you can search for that yourself. It should be easy to find something. Good luck!

Post reply on HN