Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Perforator – cluster-wide profiling tool for large data centers

github.com

Post reply on HN
Print view View original

1–10 of 16 posts

Show HN: Perforator – cluster-wide profiling tool for large data centers

#1

Post by BigRedEye »

Hey HN! We are happy to share Perforator – our internal cluster-wide profiler with great support for native languages and a built-in AutoFDO pipeline to simplify sPGO builds. Perforator allows you to profile most binaries without having to recompile or adjust the build process. We use it at Yandex to profile each pod inside a large cluster at modest speed (99Hz), collecting petabytes of profiles every day.

There's a blog post about it at https://medium.com/yandex/yandexs-high-performance-profiler-....

Inspired by Google-Wide Profiling, we started continuous profiling years ago with simple tools like poormansprofiler.org. With the rise of eBPF, we came up with a simple and elegant solution providing detailed profiles without noticeable overhead. Pretty wild when you can see the guts of your production binaries in a flamegraph without them even noticing.

Some technical details:

- Our main contribution is infrastructure for continuous PGO using AutoFDO. Google and Meta have done tremendous work on building PGO infrastructure, and we made the last missing piece of the puzzle to make this work well and scalable.

- Native binaries are profiled through eh_frame analysis, interpreted/JIT-compiled languages are profiled through perf-pid.map or hardcoded structure offsets.

- We render profiles in multiple ways, the most common one is a fast implementation of FlameGraphs, rendering 1M frames in 100ms.

- We provide Helm charts to easily deploy Perforator on your k8s cluster.

- You can use Perforator in standalone mode as a replacement for perf record.

I'd love to answer your questions about the tool!

Show HN: Perforator – cluster-wide profiling tool for large data centers
github.com

Re: Show HN: Perforator – cluster-wide profiling tool for large data centers

#5

Post by brancz »

BigRedEye wrote: post #4
brancz wrote: post #3

If I'm understanding correctly, this is collecting LBR data through hardware support for PGO/AutoFDO, right?

Yes. Although we are studying CSSPO, which uses a mixed (LBR + software-sampled stacks) approach.

I'm familiar with the paper, but it doesn't improve the situation in terms of LBR availability on cloud providers, does it?

Re: Show HN: Perforator – cluster-wide profiling tool for large data centers

#6

Post by BigRedEye »

brancz wrote: post #5
BigRedEye wrote: post #4

Earlier quoted context omitted.

Yes. Although we are studying CSSPO, which uses a mixed (LBR + software-sampled stacks) approach.

I'm familiar with the paper, but it doesn't improve the situation in terms of LBR availability on cloud providers, does it?

Yes, existing limitations apply. Without hardware LBR support, we cannot provide sPGO profiles. However, the basic profiling should work fine.

Re: Show HN: Perforator – cluster-wide profiling tool for large data centers

#7

Post by dang »

brancz wrote: post #3

If I'm understanding correctly, this is collecting LBR data through hardware support for PGO/AutoFDO, right?

(These are older comments that we merged from https://news.ycombinator.com/item?id=42888185, in case anyone was confused by the timestamps)

Re: Show HN: Perforator – cluster-wide profiling tool for large data centers

#8

Post by menaerus »

BigRedEye wrote: post #6
brancz wrote: post #5

Earlier quoted context omitted.

I'm familiar with the paper, but it doesn't improve the situation in terms of LBR availability on cloud providers, does it?

Yes, existing limitations apply. Without hardware LBR support, we cannot provide sPGO profiles. However, the basic profiling should work fine.

Blog is packed with information, thanks!

Isn't it the case that from stack traces it is rather impossible to read that function foo() is burning CPU cycles because it is memory-bound? And the reason could be rather somewhere else and not in that particular function - e.g. multiple other threads creating contention on the memory bus?

If so, doesn't this make the profile somewhat an invalid candidate for PGO?

Re: Show HN: Perforator – cluster-wide profiling tool for large data centers

#10

Post by BigRedEye »

menaerus wrote: post #8
BigRedEye wrote: post #6

Earlier quoted context omitted.

Yes, existing limitations apply. Without hardware LBR support, we cannot provide sPGO profiles. However, the basic profiling should work fine.

Blog is packed with information, thanks! Isn't it the case that from stack traces it is rather impossible to read that function foo() is burning CPU cycles because it is memory-bound? And the reason could be rather somewhere else and not in that particular function - e.g. multiple other threads creating contention on the memory bus? If so, doesn't this make the profile somewhat an invalid candidate for PGO?

It depends on the event that was sampled to generate the profiles. For example, if you sample instructions by collecting a stack trace every N instructions, you won't actually see foo() burning the CPU. However, if you look at CPU cycles, foo() will be very noticeable. Internally, we use sPGO profiles from sampling CPU cycles, not instructions.
Post reply on HN