Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Profiling with Ctrl-C

yosefk.com

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Profiling with Ctrl-C

#31

Post by kragen »

pocketarc wrote: post #25
exmadscientist wrote: post #3

My favorite hack along these lines was to put a timer/ISR on an embedded system that did nothing more than crawl up the stack frame the two or three addresses that the ISR used (yep, it was really just as dumb as [sp + 8] or whatever), and then dump that address to the serial terminal every second or so. You can fix a lot of stupid problems that way. (And most problems are stupid.) Yes, yes, a real profiler would be…

It's amusing; your 2nd paragraph drives at the core of the programmer experience - having to disclaim everything you do and say with "yes, I know there are better ways, I'm not dumb, I'm just working within constraints, and this does the job".

that's just the core of the posting-on-hn experience. when i'm programming i don't try to preemptively defuse personal attacks like that, though i do try to accurately document the advantages and disadvantages of things in my comments, which often looks like 'this is a fucking broken piece of shit that only works if none of the input bytes are null and takes cubic time'

but my motivation there isn't to keep people from calling me an idiot. that's a lost cause. it's to save me and them time rediscovering problems i already know about

Re: Profiling with Ctrl-C

#32

Post by mananaysiempre »

082349872349872 wrote: post #17
omgtehlion wrote: post #8

I mostly use GUI-based debuggers (and profilers), but even in this case I found it often useful to pause the program at random times when it appears "stuck". Most of the time I don't event need to reach for a profiler proper.

Random sampling is not only useful for quick and dirty debugging, but also for engineering nuclear bombs: https://en.wikipedia.org/wiki/Metropolis–Hastings_algorithm#...

Of course, a lot of that old neutron transport work is now used for ray tracing in cinema production. Metropolis (and MCMC is general) is one, and I also remember some sort of volumetric scattering thing as well.

Re: Profiling with Ctrl-C

#33

Post by gpderetta »

lelanthran wrote: post #16

> what do you know, there’s one billion stack frames from the nlohmann JSON parser, I guess it all gets inlined in the release build; My guess would be that it's because tail-call optimisation only happens in -O2 and above. Parsing recursively is frequently the cleanest way to implement a parser of tree-structured input, after all. If you're doing anything recursively, it makes sense to slightly restructure the recur…

Not really; knowing the library, I suspect is just many, many layers of trivial templated functions that normally just get optimized away to nothing, but at -O0 bloat the code. -Og can sometimes help in these cases.

Re: Profiling with Ctrl-C

#34

Post by bogdan-lab »

dzaima wrote: post #2

For something more systematic/reproducible, it's possible to use rr[1] to record the program, and in a replay run to the end (or whatever boundaries you care about), run "when-ticks", and do various "seek-ticks 123456789" below that number to seek to various points in the recording. I've made a thing[2] that can display that within a visual timeline (interpolated between ticks of the nearest syscalls/events, which do…

rr looks interesting. It should be useful for debugging race conditions or something "random based": once you record the issue it becomes 100% reproducible in your debugger.

Will try it next time when I have such issue. Thank you!

Re: Profiling with Ctrl-C

#35

Post by kragen »

kragen wrote: post #28
milianw wrote: post #20

Earlier quoted context omitted.

perf is easily available through yocto and buildroot (and probably other embedded linux image builders). hotspot can be downloaded as an appimage. It should not take 30-60min to set this up, but granted, learning the tools the first time always has some cost. Furthermore, note how your reasoning is quite different from what the website you linked to says - it basically says "there are no good tools" (which is untrue)…

the vast majority of embedded cpus cannot run yocto or indeed linux, even the arms but they all support gdb

(well, not all)

Re: Profiling with Ctrl-C

#36

Post by milianw »

kragen wrote: post #28
milianw wrote: post #20

Earlier quoted context omitted.

perf is easily available through yocto and buildroot (and probably other embedded linux image builders). hotspot can be downloaded as an appimage. It should not take 30-60min to set this up, but granted, learning the tools the first time always has some cost. Furthermore, note how your reasoning is quite different from what the website you linked to says - it basically says "there are no good tools" (which is untrue)…

the vast majority of embedded cpus cannot run yocto or indeed linux, even the arms but they all support gdb

True, that's another good point. But again, this reasoning is very different to the one from the linked article and website - if you have oprofile or valgrind's cachegrind available, you clearly could get perf setup instead.

I'm not debating that manual GDB sampling has its place and value. I'm debating that perf is "lying" or that it's impossible to get hold of off-CPU samples, or profiling of multithreaded code in general.

Re: Profiling with Ctrl-C

#38

Post by kragen »

milianw wrote: post #36
kragen wrote: post #28

Earlier quoted context omitted.

the vast majority of embedded cpus cannot run yocto or indeed linux, even the arms but they all support gdb

True, that's another good point. But again, this reasoning is very different to the one from the linked article and website - if you have oprofile or valgrind's cachegrind available, you clearly could get perf setup instead. I'm not debating that manual GDB sampling has its place and value. I'm debating that perf is "lying" or that it's impossible to get hold of off-CPU samples, or profiling of multithreaded code in…

yes, agreed

Re: Profiling with Ctrl-C

#39

Post by mst »

mst wrote: post #29
lelanthran wrote: post #16

> what do you know, there’s one billion stack frames from the nlohmann JSON parser, I guess it all gets inlined in the release build; My guess would be that it's because tail-call optimisation only happens in -O2 and above. Parsing recursively is frequently the cleanest way to implement a parser of tree-structured input, after all. If you're doing anything recursively, it makes sense to slightly restructure the recur…

It looks to me like you can use -foptimize-sibling-calls to get it to happen on gcc below -O2. There's also https://github.com/pietro/gcc-musttail-plugin to ensure it does happen (and clang has musttail support built in these days).

\o/ looks like that won't be needed much longer: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83324#c27

Re: Profiling with Ctrl-C

#40

Post by dooglius »

dzaima wrote: post #6
dooglius wrote: post #5

I wonder how hard it would be to have a profiler dump a big chunk of stack on each sample interrupt, convert these into core dump format, and then use gdb or whatever to decode the traces for analysis? This ought to have the touted benefits without the downside of it being slow to capture a bunch of samples.

I believe this is essentially what linux perf's "--call-graph dwarf" does. On my system that ends up producing ~33MB/s of recording data for ~4000 samples/s.

There are still issues with perf being unable to parse the debug format for some stuff, e.g. code compiled with `-ggdb3` as touched on in TFA. The idea is more: can I take one of the stacks that perf captured, and hand that off to GDB for a stack trace, without perf trying to parse/interpret it itself?
Post reply on HN