Live data from Hacker News

eBPF-based auto-instrumentation outperforms manual instrumentation

odigos.io

21–30 of 61 posts

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#21
post #7

Earlier quoted context omitted.

It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.

That’s a very big assumption, at least for Go based applications.

We also thinking on implementing fallback mechanism to automatically propagate context on the same goroutine if context.Context is not passed

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#22
post #7

Earlier quoted context omitted.

It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.

That’s a very big assumption, at least for Go based applications.

I don't think it's unreasonable, you need a Context to make a gRPC call and you get one when handling a gRPC call. It usually doesn't get lost in between.

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#23
post #20

They don't really show any of the settings they used, but for traces, I imagine if you have a reasonable sampling rate, then you aren't going to be running any code for most requests, so it won't increase latency. (Looking at their chart, I guess they are sampling .1% of requests, since 99.9% is where latency starts increasing. I am not sure if I would trace .1% of pages loads to google.com, as their table implies. R…

Thanks for the valuable feedback! We used a constant throughout of 10,000 rps. The exact testing setup can be found under “how we tested”. I think the example you gave for the lock used by Prometheus library is a great example why generation of traces/metrics is a great fit for offloading to different process (an agent). Patchyderm looks very interesting however I am not sure how you can generate distributed traces b…

> I am not sure how you can generate distributed traces based on metrics

Every log line gets an x-request-id field, and then when you combine the logs from the various components, you can see the propagation throughout our system. The request ID is a UUIDv4 but the mandatory 4 nibble in the UUIDv4 gets replaced with a digit that represents where the request came from; background task, web UI, CLI, etc. I didn't take the approach of creating a separate span ID to show sub-requests. Since you have all the logs, this extra piece of information isn't super necessary though my coworkers have asked for it a few times because every other system has it.

Since metrics are also log lines, they get the request-id, so you can do really neat things like "show me when this particular download stalled" or "show me how much bandwidth we're using from the upstream S3 server". The aggregations can take place after the fact, since you have all the raw data in the logs.

If we were running this such that we tailed the logs and sent things to Jaeger/Prometheus, a lot of this data would have to go away for cardinality reasons. But squirreling the logs away safely, and then doing analysis after the fact when a problem is suspected ends up being pretty workable. (We still do have a Prometheus exporter not based on the logs, for customers that do want alerts. For log storage, we bundle Loki.)

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#24

How hard is it to use Odigos without k8s? We mainly use docker compose for our deployments (because it's convenient, and we don't need scale), but I'm having trouble finding anything in the documentation that explains the mechanism for hooking into the container (and hence I have no clue how to repurpose it).

We are currently supporting just Kubernetes environments. docker-compose, VMs, and Serverless are on our roadmap and will be ready soon

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#25
post #13

Earlier quoted context omitted.

Our focus was on latency. The reason we were able to cut it down was due to the fact that eBPF-based automatic instrumentation separates the recording from the processing.

How did you actually reduce the latency here ?

The main factor for reduced latency is the separation between recording and processing of data. The eBPF programs are the only overhead for the instrumented process in terms of latency. The eBPF programs transfer the collected data to a separate process which handles all the exporting. In contrast to manually adding code to an application which adds latency and memory footprint in terms of handling the exported data.

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#26
post #5

How do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program see…

100%. Context propagation is _the_ key to distributed tracing, otherwise you're only seeing one side of every transaction.

I was hoping odigos was language/runtime-agnostic since it's eBPF-based, but I see it's mentioned in the repo that it only supports:

> Java, Python, .NET, Node.js, and Go

Apart from Go (that is a WIP), these are the languages already supported with Otel's (non-eBPF-based) auto-instrumentation. Apart from a win on latency (which is nice, but could in theory be combated with sampling), why else go this route?

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#27
post #26
post #5

How do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program see…

100%. Context propagation is _the_ key to distributed tracing, otherwise you're only seeing one side of every transaction. I was hoping odigos was language/runtime-agnostic since it's eBPF-based, but I see it's mentioned in the repo that it only supports: > Java, Python, .NET, Node.js, and Go Apart from Go (that is a WIP), these are the languages already supported with Otel's (non-eBPF-based) auto-instrumentation. Ap…

eBPF instrumentation does not require code changes, redeployment or restart to running applications.

We are constantly adding more language support for eBPF instrumentation and are aiming to cover the most popular programming languages soon.

Btw, not sure that sampling is really the solution to combat overhead, after all you probably do want that data. Trying to fix production issue when the data you need is missing due to sampling is not fun

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#28
post #5

How do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program see…

The eBPF programs handle passing the context through the requests by adding a field to the header as you mentioned. The injected field is according to the w3c standard.

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#29
post #7
post #5

How do you solve the context propagation issue with eBPF based instrumentation? E.g. if you get a RPC request coming in, and make an RPC request in order to serve the incoming RPC request. The traced program needs to track some ID for that request from the time it comes in, through to the place where the the HTTP request comes out. And then that ID has to get injected into a header on the wire so the next program see…

It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.

Going to be rough for supporting virtual threads then?

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#30
post #7

Earlier quoted context omitted.

It depends on the programming language being instrumented. For Go we are assuming the context.Context object is passed around between different functions or goroutines. For Java, we are using a combination of ThreadLocal tracing and Runnable tracing to support use cases like reactive and multithreaded applications.

Going to be rough for supporting virtual threads then?

We have a solution for virtual thread as well. Currently working on a blog post describing exactly how. Will update once releases
Post reply on HN