Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

All my favorite tracing tools

thume.ca

Post reply on HN
Print view View original

11–20 of 42 posts

Re: All my favorite tracing tools

#11

Post by crdavidson »

I wrote Spall, one of the lightweight profilers mentioned in the post. I loved the author's blogpost on implicit in-order forests, it was neat to see someone else's take on trees for big traces, pushed me to go way bigger than I was originally planning!

Thankfully, eytzinger-ordered 4-ary trees work totally fine at 165+ fps, even at 3+ billion functions, but I like to read back through that post once in a while just in case I hit that perf wall someday.

Working on timestamp delta-compression at the moment to pack events into much smaller spaces, and hopefully get to 10 billion in 128 GB RAM sometime soon (at least for native builds of Spall).

Thanks for the kick to keep on pushing!

Re: All my favorite tracing tools

#12

Post by kqr »

> I wanted to correlate packets with userspace events from a Python program, so I used a fun trick: Find a syscall which has an early-exit error path and bindings in most languages, and then trace calls to that which have specific arguments which produce an error.

Wow. This is some great engineering. Obviously that's what you'd do, but I'd never think of it in a thousand years!

Re: All my favorite tracing tools

#13

Post by kqr »

trishume wrote: post #9
jeffrallen wrote: post #5

The author mentions dtrace in passing. If you're into "load bearing rants", check out bcantrill's recent rant on bpftrace silently losing events and why dtrace won't do that.

I haven't actually used bpftrace myself, only BCC. I can totally imagine it being more janky than DTrace, BCC is pretty janky even if I also think it's cool. In my eBPF tracing framework I had to add special handling counters to alert you if it ever lost any events, plausible bpftrace didn't do that.

I think if you're working mostly with tracing/sampling specific applications you'll be more of a BCC person, while if you're hired to diagnose problems in a wide variety of applications then you might learn to like bpftrace more.

Re: All my favorite tracing tools

#16

Post by maccard »

criddell wrote: post #14

If you work on Windows applications, check out Event Tracing for Windows (ETW). The best place to start is Bruce Dawson’s blog: https://randomascii.wordpress.com/2015/09/24/etw-central/

If you're working with ETW traces, SuperLuminal [0] (no affiliation just a happy customer) is leaps and bounds ahead of the built-in ETW viewer.

[0] https://superluminal.eu/

Re: All my favorite tracing tools

#17

Post by RicoElectrico »

criddell wrote: post #14

If you work on Windows applications, check out Event Tracing for Windows (ETW). The best place to start is Bruce Dawson’s blog: https://randomascii.wordpress.com/2015/09/24/etw-central/

Isn't ETW a total trainwreck from a developer usability standpoint? Or so my colleagues (and the interwebs) tell me.

Re: All my favorite tracing tools

#18

Post by mark_undoio »

trishume wrote: post #3
Veserv wrote: post #2

A pretty good overview of open source solutions in the space. Missing out on one of the most useful areas for tracing which is time travel debugging. There are a number of interesting solutions there taking advantage of hardware trace, instrumentation, and deterministic replay. Even better when you get full visualization integration so you can do something like zoom in from a multiple minute trace onto a suspicious 2…

Do you know of anyone who's built that kind of time travel debugging with a trace visualization in the open outside of Javascript? I know about rr and Pernosco but don't know of trace visualization integration for either of them, that would indeed be very cool. I definitely dream of having systems like this.

At undo.io we're interested in using our time travel capability beyond conventional time travel debugging - a recording file contains everything the program did, without any advance knowledge of what you need to sample, so there's a lot of potential to get other data out of it.

I just read your post and don't think it would take much to integrate with some of the visualisations you posted about, as a first step.

We've played around in the past with a sampling profiler (code here, requires a copy of our product to be useful though it could easily port to rr): https://github.com/undoio/addons/tree/master/sample_function... which can output in a format understood by Brendan Gregg's flame frames (https://www.brendangregg.com/flamegraphs.html)

But that's not quite the kind of tracing you're talking about. We also built a printf-style interface to our recording files, which seems closer: https://docs.undo.io/PostFailureLogging.html

Something like that but outputting trace events that can be consumed by Perfetto (say) would not be so hard to add. If we considered modifying the core record/replay engine then even more powerful things become possible.

Re: All my favorite tracing tools

#19

Post by criddell »

RicoElectrico wrote: post #17
criddell wrote: post #14

If you work on Windows applications, check out Event Tracing for Windows (ETW). The best place to start is Bruce Dawson’s blog: https://randomascii.wordpress.com/2015/09/24/etw-central/

Isn't ETW a total trainwreck from a developer usability standpoint? Or so my colleagues (and the interwebs) tell me.

I don’t think so. The ability to keep drilling down deeper and deeper and the amazing sort and grouping functionality make it supremely useful.

The other tool I didn’t mention is WinDbg. In my opinion, it’s the greatest debugger on any platform.

Re: All my favorite tracing tools

#20

Post by RicoElectrico »

criddell wrote: post #19
RicoElectrico wrote: post #17

Earlier quoted context omitted.

Isn't ETW a total trainwreck from a developer usability standpoint? Or so my colleagues (and the interwebs) tell me.

I don’t think so. The ability to keep drilling down deeper and deeper and the amazing sort and grouping functionality make it supremely useful. The other tool I didn’t mention is WinDbg. In my opinion, it’s the greatest debugger on any platform.

Finally found it: https://caseymuratori.com/blog_0025
Post reply on HN