Earlier quoted context omitted.
It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.
That’s a very big assumption, at least for Go based applications.
eBPF-based auto-instrumentation outperforms manual instrumentation
21–30 of 61 posts
Re: eBPF-based auto-instrumentation outperforms manual instrumentation
#22Earlier quoted context omitted.
It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.
That’s a very big assumption, at least for Go based applications.
Re: eBPF-based auto-instrumentation outperforms manual instrumentation
#23They don't really show any of the settings they used, but for traces, I imagine if you have a reasonable sampling rate, then you aren't going to be running any code for most requests, so it won't increase latency. (Looking at their chart, I guess they are sampling .1% of requests, since 99.9% is where latency starts increasing. I am not sure if I would trace .1% of pages loads to google.com, as their table implies. R…
Thanks for the valuable feedback! We used a constant throughout of 10,000 rps. The exact testing setup can be found under “how we tested”. I think the example you gave for the lock used by Prometheus library is a great example why generation of traces/metrics is a great fit for offloading to different process (an agent). Patchyderm looks very interesting however I am not sure how you can generate distributed traces b…
Every log line gets an x-request-id field, and then when you combine the logs from the various components, you can see the propagation throughout our system. The request ID is a UUIDv4 but the mandatory 4 nibble in the UUIDv4 gets replaced with a digit that represents where the request came from; background task, web UI, CLI, etc. I didn't take the approach of creating a separate span ID to show sub-requests. Since you have all the logs, this extra piece of information isn't super necessary though my coworkers have asked for it a few times because every other system has it.
Since metrics are also log lines, they get the request-id, so you can do really neat things like "show me when this particular download stalled" or "show me how much bandwidth we're using from the upstream S3 server". The aggregations can take place after the fact, since you have all the raw data in the logs.
If we were running this such that we tailed the logs and sent things to Jaeger/Prometheus, a lot of this data would have to go away for cardinality reasons. But squirreling the logs away safely, and then doing analysis after the fact when a problem is suspected ends up being pretty workable. (We still do have a Prometheus exporter not based on the logs, for customers that do want alerts. For log storage, we bundle Loki.)
Re: eBPF-based auto-instrumentation outperforms manual instrumentation
#24How hard is it to use Odigos without k8s? We mainly use docker compose for our deployments (because it's convenient, and we don't need scale), but I'm having trouble finding anything in the documentation that explains the mechanism for hooking into the container (and hence I have no clue how to repurpose it).
Re: eBPF-based auto-instrumentation outperforms manual instrumentation
#25Earlier quoted context omitted.
Our focus was on latency. The reason we were able to cut it down was due to the fact that eBPF-based automatic instrumentation separates the recording from the processing.
How did you actually reduce the latency here ?
Re: eBPF-based auto-instrumentation outperforms manual instrumentation
#26How do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program see…
I was hoping odigos was language/runtime-agnostic since it's eBPF-based, but I see it's mentioned in the repo that it only supports:
> Java, Python, .NET, Node.js, and Go
Apart from Go (that is a WIP), these are the languages already supported with Otel's (non-eBPF-based) auto-instrumentation. Apart from a win on latency (which is nice, but could in theory be combated with sampling), why else go this route?
Re: eBPF-based auto-instrumentation outperforms manual instrumentation
#27How do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program see…
100%. Context propagation is _the_ key to distributed tracing, otherwise you're only seeing one side of every transaction. I was hoping odigos was language/runtime-agnostic since it's eBPF-based, but I see it's mentioned in the repo that it only supports: > Java, Python, .NET, Node.js, and Go Apart from Go (that is a WIP), these are the languages already supported with Otel's (non-eBPF-based) auto-instrumentation. Ap…
We are constantly adding more language support for eBPF instrumentation and are aiming to cover the most popular programming languages soon.
Btw, not sure that sampling is really the solution to combat overhead, after all you probably do want that data. Trying to fix production issue when the data you need is missing due to sampling is not fun
Re: eBPF-based auto-instrumentation outperforms manual instrumentation
#28How do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program see…
Re: eBPF-based auto-instrumentation outperforms manual instrumentation
#29How do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program see…
It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.
Re: eBPF-based auto-instrumentation outperforms manual instrumentation
#30Earlier quoted context omitted.
It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.
Going to be rough for supporting virtual threads then?