Of course it outperforms it, but it's basic instrumentation, how do you properly select the labels for example? In your application you will have custom instrumentation for business logic, so what do you do? Now you have two systems instrumenting the same app?
You can enrich the spans created by eBPF by using OpenTelemetry APIs as usual, the eBPF instrumentation is a replacement for the instrumentation SDK. The eBPF program will detect the data recorded via the APIs and will add it to the final trace combining both automatic and manually created data.
Anyone from the dtrace community want to enlighten a n00b about how eBPF compares to what dtrace does?
I don’t have a lot of experience using dtrace, but AFAIK the big advantage of eBPF over dtrace is that you do not need to instrument your application with static probes during coding.
DTrace (on Solaris at least) can instrument any userspace symbol or address, no need for static tracepoints in the app.
One problem that DTrace has is that the "pid" provider that you use for userspace app tracing only works on processes that are already running. So, if more processes with the executable of interest launch after you've started DTrace, its pid provider won't catch the new ones. Then you end up doing some tricks like tracking exec-s of the binary and restarting your DTrace script...
Anyone from the dtrace community want to enlighten a n00b about how eBPF compares to what dtrace does?
They're really very different -- with very different origins and constraints. If you want to hear about my own experiences with bpftrace, I got into this a bit recently.[0] (And in fact, one of my questions about the article is how they deal with silently dropped data in eBPF -- which I found to be pretty maddening.)
Anyone from the dtrace community want to enlighten a n00b about how eBPF compares to what dtrace does?
They're really very different -- with very different origins and constraints. If you want to hear about my own experiences with bpftrace, I got into this a bit recently.[0] (And in fact, one of my questions about the article is how they deal with silently dropped data in eBPF -- which I found to be pretty maddening.) [0] https://www.youtube.com/watch?v=mqvVmYhclAg#t=12m0s
By dropped data do you mean by exceeding the size of the allocated ring buffer/perf buffer? If so this is configurable by the user, so you can adjust is according to the expected load
I don’t have a lot of experience using dtrace, but AFAIK the big advantage of eBPF over dtrace is that you do not need to instrument your application with static probes during coding.
DTrace (on Solaris at least) can instrument any userspace symbol or address, no need for static tracepoints in the app. One problem that DTrace has is that the "pid" provider that you use for userspace app tracing only works on processes that are already running. So, if more processes with the executable of interest launch after you've started DTrace, its pid provider won't catch the new ones. Then you end up doing s…
That's not exactly correct, and is merely a consequence of the fact that you are trying to use the pid provider. The issue that you're seeing is that pid probes are created on-the-fly -- and if you don't demand that they are created in a new process, they in fact won't be. USDT probes generally don't have this issue (unless they are explicitly lazily created -- and some are). So you don't actually need/want to restart your DTrace script, you just want to force probes to be created in new processes (which will necessitate some tricks, just different ones).
They're really very different -- with very different origins and constraints. If you want to hear about my own experiences with bpftrace, I got into this a bit recently.[0] (And in fact, one of my questions about the article is how they deal with silently dropped data in eBPF -- which I found to be pretty maddening.) [0] https://www.youtube.com/watch?v=mqvVmYhclAg#t=12m0s
By dropped data do you mean by exceeding the size of the allocated ring buffer/perf buffer? If so this is configurable by the user, so you can adjust is according to the expected load
eBPF can drop data silently under quite a few conditions, unfortunately. And -- most frustratingly -- it's silent, so it's not even entirely clear which condition you've fallen into. This alone is a pretty significant with respect to DTrace: when/where DTrace drops data, there is always an indicator as to why. And to be clear, this isn't a difference merely of implementation (though that too, certainly), but of principle: DTrace, at root, is a debugger -- and it strives to be as transparent to the user as possible as to the truth of the underlying system.
Anyone from the dtrace community want to enlighten a n00b about how eBPF compares to what dtrace does?
They're really very different -- with very different origins and constraints. If you want to hear about my own experiences with bpftrace, I got into this a bit recently.[0] (And in fact, one of my questions about the article is how they deal with silently dropped data in eBPF -- which I found to be pretty maddening.) [0] https://www.youtube.com/watch?v=mqvVmYhclAg#t=12m0s
I listened to this live! That's probably why I was wondering, because I remember you talking about something you used in Linux that didn't quite live up to your expectations with DTrace, but I didn't catch all of the names. Thanks!
The column in the table claiming the "number of page loads that would experience the 99th %ile" is mathematically suspect. It directly contradicts what a percentile is. By definition, at 99th percentile, if I have 100 page loads, the one with the worst latency would be over the 99th percentile. That's not 85.2%, 87.1%, 67.6%, etc. The formula shown in that column makes no sense at all.
That's not what that column is supposed to mean afaict. The way I read it is it's showing that if the website requires hundreds of different parallel backend service calls to serve the page load, what's the probability a page load hits the p99 instrumentation latency? We have a similar chart at my job to illustrate the point that high p99 latency on a backend service doesn't mean only 1% of end-user page loads are af…
Ah, I see. So, for example, if one page request would result in 190 different backend requests to fulfill, then the possibility that at least one of those subrequests exceeds the 99th percentile would be 85.2%. That makes a lot more sense.
That’s a very big assumption, at least for Go based applications.
I don't think it's unreasonable, you need a Context to make a gRPC call and you get one when handling a gRPC call. It usually doesn't get lost in between.
True for gRPC, but not necessarily for HTTP - the HTTP client and server packages that ship with Go predate the Context package by quite a long while.