Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
blog.janestreet.com
Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
1–10 of 20 posts
Re: Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
#2Oh man, this is major. I would’ve loved to have something like that 10 years ago when CPU was a bit more precious. Still very useful today, just not to the same extent.
Re: Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
#3Oh man, this is major. I would’ve loved to have something like that 10 years ago when CPU was a bit more precious. Still very useful today, just not to the same extent.
I wish more devs continued to realize that CPU is precious. One executable running in isolation is only limited by how fast it can consume clock cycles, but we now have hundreds of executables each running dozens or hundreds of functions all the time. One unoptimized IO function can cause stutters across the whole system as it runs. Thousands of unoptimized IO functions can....
Well, we all know. We use systems like that daily.
Re: Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
#4Oh man, this is major. I would’ve loved to have something like that 10 years ago when CPU was a bit more precious. Still very useful today, just not to the same extent.
Didnt know this existed, interesting, but certainly could be useful at a forensic level, have had tools to highlight slow running multi threaded code in apps for probably about 15years now, but this takes it to a whole new level.
From the link it says: " it needs a post-Skylake Intel processor " https://en.wikipedia.org/wiki/Skylake_(microarchitecture)
Man page has a description https://www.man7.org/linux/man-pages/man1/perf-intel-pt.1.ht...
Dont know who LauterBach in Germany are, but they have a training manual which goes back to 1989.
https://www2.lauterbach.com/pdf/training_ipt_trace.pdf
https://www2.lauterbach.com/pdf/trace_intel_pt.pdf
https://www2.lauterbach.com/pdf/debugger_x86.pdf
From their manuals "The Intel® Processor Trace (IPT) works similar to the LBR and BTS feature of Ix86 based cores (see “CPU specific Onchip Trace Commands” (debugger_x86.pdf)"
I knew about the debugger on ARM cpu's like the Rpi, didnt know about Intel having one, but its suggested AMD dont have one either, so there might be some security reason for that, depends on if the trace is just output only or whether its possible to use things like SOIC clips to alter bits and bytes in realtime, but like slowed down not normal cpu clock speeds.
Re: Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
#5The visualization tools presented look really nice, but they seem to present program execution as sequential and linear, which is a model that seems like it will really break down at these time scales (10s of cycles).
Modern processors will look hundreds of instructions into the future and try to start executing them as soon as possible. Branches are predicted far in advance of when they can actually be evaluated. Many instructions can be executing simultaneously. A clean tidy flame graph showing 1-3ns slices (~5 cycles) cannot help but be a vast simplification of what the CPU is really doing.
The linked page about Processor Trace says this:
> instruction data (control flow) is perfectly accurate but timing information is less accurate
The article mentions using magic-trace to detect changes in inlining decisions made by the compiler. This is a case where it will shine, since PT can perfectly capture the control flow, and it doesn't necessarily rely on having perfect timestamps for everything.
Re: Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
#6[dead]
Re: Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
#7The visualization tools presented look really nice, but they seem to present program execution as sequential and linear, which is a model that seems like it will really break down at these time scales (10s of cycles). Modern processors will look hundreds of instructions into the future and try to start executing them as soon as possible. Branches are predicted far in advance of when they can actually be evaluated. Ma…
> A clean tidy flame graph showing 1-3ns slices (~5 cycles) cannot help but be a vast simplification of what the CPU is really doing.
Sure, the thinnest slices on the highest zoom are going to be misleading. They're also not what you generally will want to be looking at (though they may provide context for you to identify the part higher functions taking a long time or hints about cache contention etc).
Re: Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
#8The best I've found until now is gperftools (In contrast to perf you get good results even with -O3 optimization and heavily inlined code). This seems to be much more accurate, but I'm not sure we can handle that amount of data because we usually take longer traces.
Re: Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
#9Oh man, this is major. I would’ve loved to have something like that 10 years ago when CPU was a bit more precious. Still very useful today, just not to the same extent.
Didnt know this existed, interesting, but certainly could be useful at a forensic level, have had tools to highlight slow running multi threaded code in apps for probably about 15years now, but this takes it to a whole new level. From the link it says: " it needs a post-Skylake Intel processor " https://en.wikipedia.org/wiki/Skylake_(microarchitecture) Man page has a description https://www.man7.org/linux/man-pages/m…
>Dont know who LauterBach in Germany are
They make a lot of debuggers for embedded targets. Often with tracing capability and such. Really good tools.
Re: Magic-trace: Diagnose tricky performance issues with Intel Processor Trace
#10tracy (https://github.com/wolfpld/tracy), mentioned in this article as well, for some reason is criminally underused, unknown etc. by wider community.