Speaking of keyboard shortcuts, I miss BSD's Ctrl-T and SIGINFO. It often helped to see if a process was hung.
Profiling with Ctrl-C
11–20 of 42 posts
Re: Profiling with Ctrl-C
#12Re: Profiling with Ctrl-C
#13Speaking of keyboard shortcuts, I miss BSD's Ctrl-T and SIGINFO. It often helped to see if a process was hung.
I don't know exactly what these BSD things did, but there is a super easy way nowadays to get the stack for any process:
eu-stack -i -p $(pidof ...)
Re: Profiling with Ctrl-C
#14I wonder how hard it would be to have a profiler dump a big chunk of stack on each sample interrupt, convert these into core dump format, and then use gdb or whatever to decode the traces for analysis? This ought to have the touted benefits without the downside of it being slow to capture a bunch of samples.
I believe this is essentially what linux perf's "--call-graph dwarf" does. On my system that ends up producing ~33MB/s of recording data for ~4000 samples/s.
With `-z` (zstd compression) you can bring down the disk-space cost of dwarf unwinding by a factor of ~100 based on my personal experience.
to GP: What you describe sounds like https://github.com/koute/not-perf to me
Re: Profiling with Ctrl-C
#15https://poormansprofiler.org/
The premise of this website and articles like https://yosefk.com/blog/how-profilers-lie-the-cases-of-gprof... just show that the authors are using the wrong tools. It is nowadays relatively easy to also look at off-CPU time when profiling with perf (e.g. https://github.com/KDAB/hotspot/?tab=readme-ov-file#off-cpu-...). The idea is to use sampling for the on-CPU periods and then combine that with the off-CPU time measured between context switches. VTune also supported this mode for many years.
Re: Profiling with Ctrl-C
#16> what do you know, there’s one billion stack frames from the nlohmann JSON parser, I guess it all gets inlined in the release build;
My guess would be that it's because tail-call optimisation only happens in -O2 and above.
Parsing recursively is frequently the cleanest way to implement a parser of tree-structured input, after all.
If you're doing anything recursively, it makes sense to slightly restructure the recursive call to be the last call in the scope, so that TCO can be applied.
Re: Profiling with Ctrl-C
#17I mostly use GUI-based debuggers (and profilers), but even in this case I found it often useful to pause the program at random times when it appears "stuck". Most of the time I don't event need to reach for a profiler proper.
Random sampling is not only useful for quick and dirty debugging, but also for engineering nuclear bombs: https://en.wikipedia.org/wiki/Metropolis–Hastings_algorithm#...
Re: Profiling with Ctrl-C
#18My favorite hack along these lines was to put a timer/ISR on an embedded system that did nothing more than crawl up the stack frame the two or three addresses that the ISR used (yep, it was really just as dumb as [sp + 8] or whatever), and then dump that address to the serial terminal every second or so. You can fix a lot of stupid problems that way. (And most problems are stupid.) Yes, yes, a real profiler would be…
> My favorite hack along these lines was to put a timer/ISR on an embedded system that did nothing more than crawl up the stack frame the two or three addresses that the ISR used (yep, it was really just as dumb as [sp + 8] or whatever), and then dump that address to the serial terminal every second or so.
Did the same, but added a stack canary to tell if I overflowed, and wrote my call-stack results to a hardcoded address at the very end of memory (last 32 bytes, IIRC). When the chip reset, in addition to the reset flags (brownout, etc), I could peek that memory to see what the last addresses in the callstack were.
Helped immensely in figuring out a transient bug (device resets) which was hard to repro.
Re: Profiling with Ctrl-C
#19https://poormansprofiler.org/
The premise of this website and articles like https://yosefk.com/blog/how-profilers-lie-the-cases-of-gprof... just show that the authors are using the wrong tools. It is nowadays relatively easy to also look at off-CPU time when profiling with perf (e.g. https://github.com/KDAB/hotspot/?tab=readme-ov-file#off-cpu-... ). The idea is to use sampling for the on-CPU periods and then combine that with the off-CPU time mea…
> The premise of this website and articles like https://yosefk.com/blog/how-profilers-lie-the-cases-of-gprof... just show that the authors are using the wrong tools. It is nowadays relatively easy to also look at off-CPU time when profiling with perf (e.g. https://github.com/KDAB/hotspot/?tab=readme-ov-file#off-cpu-...).
I think, firstly, that spending 15s trying the CTRL-c approach is a worthwhile tradeoff. If you don't find anything, then sure, spend another 30m - 60m setting up perf, KDAB, etc. Maybe more if you're on an embedded device.
Secondly, the author seems to say that he's used this on embedded devices with no output but a serial line for the debugger. This is also a 15s effort[1].
It's basically a very low effort task, takes seconds to determine if it worked or not, and if it doesn't work you've only lost a few seconds.
[1] I'm assuming that if you're developing on a device supporting a serial GDB connection, you've already got the debugger working.
Re: Profiling with Ctrl-C
#20Earlier quoted context omitted.
The premise of this website and articles like https://yosefk.com/blog/how-profilers-lie-the-cases-of-gprof... just show that the authors are using the wrong tools. It is nowadays relatively easy to also look at off-CPU time when profiling with perf (e.g. https://github.com/KDAB/hotspot/?tab=readme-ov-file#off-cpu-... ). The idea is to use sampling for the on-CPU periods and then combine that with the off-CPU time mea…
> The premise of this website and articles like https://yosefk.com/blog/how-profilers-lie-the-cases-of-gprof ... just show that the authors are using the wrong tools. It is nowadays relatively easy to also look at off-CPU time when profiling with perf (e.g. https://github.com/KDAB/hotspot/?tab=readme-ov-file#off-cpu- ...). I think, firstly, that spending 15s trying the CTRL-c approach is a worthwhile tradeoff. If you…
perf is easily available through yocto and buildroot (and probably other embedded linux image builders). hotspot can be downloaded as an appimage. It should not take 30-60min to set this up, but granted, learning the tools the first time always has some cost.
Furthermore, note how your reasoning is quite different from what the website you linked to says - it basically says "there are no good tools" (which is untrue) whereas you are saying "manual GDB sampling might be good enough and is easier to setup than a good tool" (which is certainly true).