Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

eBPF-based auto-instrumentation outperforms manual instrumentation

odigos.io

Post reply on HN
Print view View original

51–60 of 61 posts

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#51

Post by jetbalsa »

edenfed wrote: post #33
rocmcd wrote: post #31

Earlier quoted context omitted.

All good points, thank you. What's the limit on language support? Is it theoretically possible to support any language/runtime? Or does it come down to the protocol (HTTP, gRPC, etc) being used by the communicating processes?

We already solved compiled languages (Go, C, Rust) and JIT languages (Java, C#). Interpreted languages (Python, JS) are the only ones left, hopefully we will solve these as well soon. The big challenge is supporting all the different runtimes, once that is solved implementing support for different protocols / open-source libraries is not as complicated.

Got to get PHP on that list :)

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#52

Post by marwis »

edenfed wrote: post #8
CSDude wrote: post #4

Somewhat related, I mainly code in Kotlin. Adding open telemetry was just adding agent to command line args (usual Java/JVM magic most people don't like). Then I had a project in Go and I got so tired of all the steps it took (setup and ensuring each context is instrumented) and just gave up. We still add our manual instrumentation for customization, but auto-instrumentation made adoption much easier in the day 0.

I think eBPF has also great potential to help JVM-based languages. Especially around performance aspects even comparing to the current java agents which use bytecode manipulation.

The article mentions avoiding GC pressure and separation between recording and processing as big wins for performance for runtimes like Java but you could do the same inside Java by using ring buffer, no?

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#53

Post by edenfed »

marwis wrote: post #52
edenfed wrote: post #8

Earlier quoted context omitted.

I think eBPF has also great potential to help JVM-based languages. Especially around performance aspects even comparing to the current java agents which use bytecode manipulation.

The article mentions avoiding GC pressure and separation between recording and processing as big wins for performance for runtimes like Java but you could do the same inside Java by using ring buffer, no?

Interesting idea. I think that as long as you able to do processing, serializing and delivery in other process and save this work from your application runtime you should see great performance

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#54

Post by tanelpoder »

bcantrill wrote: post #46
tanelpoder wrote: post #43

Earlier quoted context omitted.

DTrace (on Solaris at least) can instrument any userspace symbol or address, no need for static tracepoints in the app. One problem that DTrace has is that the "pid" provider that you use for userspace app tracing only works on processes that are already running. So, if more processes with the executable of interest launch after you've started DTrace, its pid provider won't catch the new ones. Then you end up doing s…

That's not exactly correct, and is merely a consequence of the fact that you are trying to use the pid provider. The issue that you're seeing is that pid probes are created on-the-fly -- and if you don't demand that they are created in a new process, they in fact won't be. USDT probes generally don't have this issue (unless they are explicitly lazily created -- and some are). So you don't actually need/want to restar…

So how would you demand that they’d be created in a new process? I was already using pid* provider years ago when I was working on this (and wasn’t using static compiled-in tracepoints).

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#55

Post by chii »

RonFeder wrote: post #25
grazio wrote: post #13

Earlier quoted context omitted.

How did you actually reduce the latency here ?

The main factor for reduced latency is the separation between recording and processing of data. The eBPF programs are the only overhead for the instrumented process in terms of latency. The eBPF programs transfer the collected data to a separate process which handles all the exporting. In contrast to manually adding code to an application which adds latency and memory footprint in terms of handling the exported data.

but the processing will still cost CPU time which takes it away from the 'main' process, unless it's transferred away from the machine and processed elsewhere. Unless if eBPF can do such processing much more efficiently than the application's own code, i don't see how it reduces latency differently from a properly threaded app. Of course, using eBPF makes an app instrument-able without changes is good enough a reason to use it.

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#56

Post by remoquete »

CSDude wrote: post #4

Somewhat related, I mainly code in Kotlin. Adding open telemetry was just adding agent to command line args (usual Java/JVM magic most people don't like). Then I had a project in Go and I got so tired of all the steps it took (setup and ensuring each context is instrumented) and just gave up. We still add our manual instrumentation for customization, but auto-instrumentation made adoption much easier in the day 0.

OTel autoinstrumentation is in the works, check it out: https://github.com/open-telemetry/opentelemetry-go-instrumen... (I wrote and tested that guide).

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#57

Post by nikolay_sivko »

Disclaimer: I'm a co-founder of Coroot. We're currently benchmarking our eBPF-based agent to measure its performance impact.

Could you please elaborate on a few more details about your benchmark?

- Did you measure the CPU usage of the eBPF agent?

- How does Odigos handle eBPF's perfmap overflow, and did you measure any lost events between the kernel and the agent?

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#58

Post by RonFeder »

chii wrote: post #55
RonFeder wrote: post #25

Earlier quoted context omitted.

The main factor for reduced latency is the separation between recording and processing of data. The eBPF programs are the only overhead for the instrumented process in terms of latency. The eBPF programs transfer the collected data to a separate process which handles all the exporting. In contrast to manually adding code to an application which adds latency and memory footprint in terms of handling the exported data.

but the processing will still cost CPU time which takes it away from the 'main' process, unless it's transferred away from the machine and processed elsewhere. Unless if eBPF can do such processing much more efficiently than the application's own code, i don't see how it reduces latency differently from a properly threaded app. Of course, using eBPF makes an app instrument-able without changes is good enough a reason…

Compared to multi-threaded process, there is still a big advantage in terms of latency. Handling all the exporting in the same process will greatly effect GC operations which require stop the world handling.

Re: eBPF-based auto-instrumentation outperforms manual instrumentation

#59

Post by anuraaga »

In the age of supply chain attack weariness and general risk skyrocketing, it is a bit funny seeing the many observability vendors wanting you to give them kernel mode access. And it's sad that most apps that will be most in need of automatic instrumentation are "frozen" rarely developed / updated apps at critical institutions like banks.

As for the original post, opentelemetry is forced to be relatively slow because of a huge amount of semantic conventions that are meant to make data more useful. I won't go into the legitimacy of that, but while I haven't been able to verify the data this solution records, it is very unlikely to be recording as much information. Manual instrumentation would never loose to eBPF in principle, at least in a compiled language like Go, but eBPF does have great potential to perform better than OTel while recording far less data. Then comes blog post, users giving the keys to their kernel, and data ending up in the hands of an enemy state. I doubt that's the case this time but it's only a matter of time.

Banking apps if you see this, please just instrument your code. Thank you.

Post reply on HN