Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Magic-trace – High-resolution traces of what a process is doing

github.com

Post reply on HN
Print view View original

131–140 of 146 posts

Re: Magic-trace – High-resolution traces of what a process is doing

#131

Post by xiphias2 »

dchftcs wrote: post #128
xiphias2 wrote: post #127

,,magic-trace.org is a fork of Perfetto, with minor modifications. We'd like to thank the people at Google responsible for it. It's a high quality codebase that solves a hard problem well.'' https://perfetto.dev I don't like that this comment is hidden on the bottom of the page, as my first impression of the page was that the work of creating the high frequency trace was done by Jane Street (I don't like it in the ot…

The trace-producing part of magic-trace is built on top of the more primitive Intel Processing Trace feature, not perfetto. It's still standing on the shoulder of a giant for sure, but few people would be able to effectively utilize Intel Processing Trace without the ergonomic improvements. perfetto is "just" the profiler UI.

Thanks, it wasn’t clear what was the modification, now it makes more sense.

Re: Magic-trace – High-resolution traces of what a process is doing

#132

Post by shaklee3 »

mirekrusin wrote: post #92
b20000 wrote: post #84

Earlier quoted context omitted.

I doubt that it is used for actual trading. Maybe the OP can elaborate what the live infrastructure and backtesting setup look like?

Of course they use it for trading and everything around for many years.

they use it for the control plane. it's clearly not used for actually submitting trades.

Re: Magic-trace – High-resolution traces of what a process is doing

#133

Post by throw20220422 »

shaklee3 wrote: post #132
mirekrusin wrote: post #92

Earlier quoted context omitted.

Of course they use it for trading and everything around for many years.

they use it for the control plane. it's clearly not used for actually submitting trades.

Do you mean this tool, or OCaml? They are definitely writing their actual trading systems in OCaml, they have been publicizing this for many years.

Re: Magic-trace – High-resolution traces of what a process is doing

#135

Post by wyldfire »

So instead of sampling, or hw perf counters, IPT does tracing? perf counters are able to attribute cycles or cache misses to instructions. but only in aggregate. If your program isn't a 100% computation, cycles consumed won't necessarily point to the bottleneck. So IIUC IPT can tell you more than just the statistics but instead can tell the real story - the sequence of instructions executed? If so, I can see this painting a much clearer picture. But what are the limitations? some small buffers that overflow after a few million instructions or pipeline flushes? If the IPT buffer(s) overflow does magic-trace indicate gaps?

Is it possible to combine PMU or sampling with IPT to get multiple profiling dimensions in the same run? Not just what sequence of instructions were executed but where in time-and-code the branch mispredictions, cache misses, etc. occurred?

Re: Magic-trace – High-resolution traces of what a process is doing

#136

Post by brendangregg »

Thanks, good to see a new perf tool (and not just another procfs top clone :-). I've started using processor trace in the cloud, thanks to bare-metal instances, but for years I couldn't touch it (not available in VMs). There's a wealth of new information it provides, and we need better tooling on top of it, like magic-trace.

Glad to see "overhead" mentioned and quantified. I'd put the 2-10% at the top though, as that's heavy handed for some environments (can trigger a production fail-over).

I see magic-trace has implemented what some call "flame charts" (time on the x-axis) and not "flame graphs" (alphabet on the x-axis). The best tools do both (e.g., TraceCompass). Please do both! Will make seeing the big picture easy (flame graphs) and then zooming into time-based patterns easy too (flame charts).

Re: Magic-trace – High-resolution traces of what a process is doing

#137

Post by macintux »

ben0x539 wrote: post #108
haggy wrote: post #34

Earlier quoted context omitted.

The magic of PR's :)

Are people generally cool with unsolicited PRs to "editorial" content? I'd have assumed that stating or not stating something in a README is mostly an intentional decision.

I doubt there's a universal answer to that question, but I think most projects tend to overlook important information because the participants have been too deep in the weeds for too long. You tend to forget what newcomers won't find obvious about your tool.

Re: Magic-trace – High-resolution traces of what a process is doing

#138

Post by cgaebel »

wyldfire wrote: post #135

So instead of sampling, or hw perf counters, IPT does tracing? perf counters are able to attribute cycles or cache misses to instructions. but only in aggregate. If your program isn't a 100% computation, cycles consumed won't necessarily point to the bottleneck. So IIUC IPT can tell you more than just the statistics but instead can tell the real story - the sequence of instructions executed? If so, I can see this pai…

You basically understand how this works; you see everything, but there might be gaps in the trace. In our experience they're rare (In fact, if you look carefully at the demo gifs in the README, that trace had 5 decode errors! Nonetheless, it was extremely usable.

Snapshot sizes are configurable--you can go back as far as you like. However, the trace viewer tends to crash when the trace files reach the hundreds of MB and you'll need to do some work to set up a trace processor outside of your browser for the UI to connect to. The UI will offer up some docs if you actually run into this.

I'm so glad you asked us about PMU events, we've been thinking a lot about those. These are available in traces of the efficiency cores of Alder Lake CPUs, but nothing else. When we get our hands on a server class part with PMU tracing we'll add support ASAP. We conjecture that it will be absurdly useful to see cache events on a timeline next to call stacks.

Re: Magic-trace – High-resolution traces of what a process is doing

#139

Post by cgaebel »

brendangregg wrote: post #136

Thanks, good to see a new perf tool (and not just another procfs top clone :-). I've started using processor trace in the cloud, thanks to bare-metal instances, but for years I couldn't touch it (not available in VMs). There's a wealth of new information it provides, and we need better tooling on top of it, like magic-trace. Glad to see "overhead" mentioned and quantified. I'd put the 2-10% at the top though, as that…

We are similarly sad about how unavailable Intel PT is in VMs. In 2022, being unavailable on Macs and VMs raises the barrier to entry extraordinarily high for many people in our target audience. Not sure if working outside of work is your cup of tea, but we've found Intel NUCs to be Good point about overhead. I've moved the 2%-10% number front and center, and wrote up a bit more detail about where that comes from in a new wiki page: https://github.com/janestreet/magic-trace/wiki/Overhead

We'll think about adding flame graphs. We unfortunately have little experience writing responsive web UIs, the excellent Perfetto developers did all of the heavy lifting on that front. But who knows, maybe an enterprising Open Source Contributor could help us out. I see Matt Godbolt was asking questions in their discord the other day...

Re: Magic-trace – High-resolution traces of what a process is doing

#140

Post by [deleted] »

isogon wrote: post #97
kolbe wrote: post #94

Earlier quoted context omitted.

Do you know this for a fact? I've done some work in the industry where I needed to make fast software, but never the like sub-microsecond tick-to-trade type fast, so I really don't know. There was a great presentation from 2017 about some of Optiver's low latency techniques[1]. I had assumed they released it because the had obviated all of them by switching to FPGAs, but I don't know. Either way, he suggested that if…

The idea is precisely that you want to avoid pinging main memory at all, which is possible (in the happy case) if you do things correctly with DDIO. Not everything is done in hardware where I am. I am wary of saying much because my employer frowns on it, and admittedly I work on the software more than the hardware, but DDIO is certainly important to us.

[deleted]
Post reply on HN