eBPF-based auto-instrumentation outperforms manual instrumentation

They don't really show any of the settings they used, but for traces, I imagine if you have a reasonable sampling rate, then you aren't going to be running any code for most requests, so it won't increase latency. (Looking at their chart, I guess they are sampling .1% of requests, since 99.9% is where latency starts increasing. I am not sure if I would trace .1% of pages loads to google.com, as their table implies. Rather, I'd pick something like 1 request per second, so that latency does not increase as load increases.)

A lot of Go metrics libraries, specifically Prometheus, introduce a lot of lock contention around incrementing metrics. This was unacceptably slow for our use case at work and I ended up writing a metrics system that doesn't take any locks for most cases.

(There is the option to introduce a lock for metrics that are emitted on a timed basis; i.e. emit tx_bytes every 10s or 1MiB instead of at every Write() call. But this lock is not global to the program; it's unique to the metric and key=value "fields" on the metric. So you can have a lot of metrics around and not content on locks.)

The metrics are then written to the log, which can be processed in real time to synthesize distributed traces and prometheus metrics, if you really want them: https://github.com/pachyderm/pachyderm/blob/master/src/inter... (Our software is self-hosted, and people don't have those systems set up, so we mostly consume metrics/traces in log form. When customers have problems, we prepare a debug bundle that is mostly just logs, and then we can further analyze the logs on our side to see event traces, metrics, etc.)

As for eBPF, that's something I've wanted to use to enrich logs with more system-level information, but most customers that run our software in production aren't allowed to run anything as root, and thus eBPF is unavailable to them. People will tolerate it for things like Cilium or whatever, but not for ordinary applications that users buy and request that their production team install for them. Production Linux at big companies is super locked down, it seems, much to my disappointment. (Personally, my threat model for Linux is that if you are running code on the machine, you probably have root through some yet-undiscovered kernel bug. Historically, I've been right. But that is not the big companies' security teams' mental model, it appears. They aren't paranoid enough to run each k8s pod in a hypervisor, but are paranoid enough to prevent using CAP_SYS_ADMIN or root.)

The column in the table claiming the "number of page loads that would experience the 99th %ile" is mathematically suspect. It directly contradicts what a percentile is.

By definition, at 99th percentile, if I have 100 page loads, the one with the worst latency would be over the 99th percentile. That's not 85.2%, 87.1%, 67.6%, etc. The formula shown in that column makes no sense at all.

I recommend watching Gil Tene’s talk, I think he explains the math better than I do: https://www.youtube.com/watch?v=lJ8ydIuPFeU

That's not what that column is supposed to mean afaict. The way I read it is it's showing that if the website requires hundreds of different parallel backend service calls to serve the page load, what's the probability a page load hits the p99 instrumentation latency?

We have a similar chart at my job to illustrate the point that high p99 latency on a backend service doesn't mean only 1% of end-user page loads are affected.

Thanks for the valuable feedback! We used a constant throughout of 10,000 rps. The exact testing setup can be found under “how we tested”.

I think the example you gave for the lock used by Prometheus library is a great example why generation of traces/metrics is a great fit for offloading to different process (an agent).

Patchyderm looks very interesting however I am not sure how you can generate distributed traces based on metrics, how do you fill in the missing context propagation?

Our way to deal with eBPF root requirements is to be transparent as possible. This is why we donated the code to the CNCF and developing as part of the OpenTelemetry community. We hope that being open will make users trust us. You can see the relevant code here: https://github.com/open-telemetry/opentelemetry-go-instrumen...