Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenTelemetry for Go: Measuring overhead costs

coroot.com

Post reply on HN
Print view View original

11–20 of 51 posts

Re: OpenTelemetry for Go: Measuring overhead costs

#11

Post by nikolay_sivko »

vanschelven wrote: post #9

The article never really explains what eBPF is -- AFAIU, it’s a kernel feature that lets you trace syscalls and network events without touching your app code. Low overhead, good for metrics, but not exactly transparent. It’s the umpteenth OTEL-critical article on the front page of HN this month alone... I have to say I share the sentiment but probably for different reasons. My take is quite the opposite: most value i…

I'm the author. I wouldn’t say the post is critical of OTEL. I just wanted to measure the overhead, that’s all. Benchmarks shouldn’t be seen as critique. Quite the opposite, we can only improve things if we’ve measured them first.

Re: OpenTelemetry for Go: Measuring overhead costs

#12

Post by politician »

vanschelven wrote: post #9

The article never really explains what eBPF is -- AFAIU, it’s a kernel feature that lets you trace syscalls and network events without touching your app code. Low overhead, good for metrics, but not exactly transparent. It’s the umpteenth OTEL-critical article on the front page of HN this month alone... I have to say I share the sentiment but probably for different reasons. My take is quite the opposite: most value i…

I don't want to take away from your point, and yet... if anyone lacks background knowledge these days the relevant context is just an LLM prompt away.

Re: OpenTelemetry for Go: Measuring overhead costs

#13

Post by orochimaaru »

Thaxll wrote: post #4

Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.

Metrics are usually minimal overheard. Traces need to be sampled. Logs need to be sampled at error/critical levels. You also need to be able to dynamically change sampling and log levels.

100% traces are a mess. I didn’t see where he setup sampling.

Re: OpenTelemetry for Go: Measuring overhead costs

#14

Post by vanschelven »

politician wrote: post #12
vanschelven wrote: post #9

The article never really explains what eBPF is -- AFAIU, it’s a kernel feature that lets you trace syscalls and network events without touching your app code. Low overhead, good for metrics, but not exactly transparent. It’s the umpteenth OTEL-critical article on the front page of HN this month alone... I have to say I share the sentiment but probably for different reasons. My take is quite the opposite: most value i…

I don't want to take away from your point, and yet... if anyone lacks background knowledge these days the relevant context is just an LLM prompt away.

It was always "a search away" but on the _web_ one might as well use... A hyperlink

Re: OpenTelemetry for Go: Measuring overhead costs

#16

Post by malkia »

sa46 wrote: post #10

Funny timing—I tried optimizing the Otel Go SDK a few weeks ago ( https://github.com/open-telemetry/opentelemetry-go/issues/67... ). I suspect you could make the tracing SDK 2x faster with some cleverness. The main tricks are: - Use a faster time.Now(). Go does a fair bit of work to convert to the Go epoch. - Use atomics instead of a mutex. I sent a PR, but the reviewer caught correctness issues. Atomics are subtle a…

There is an effort to use arrow format for metrics too - https://github.com/open-telemetry/otel-arrow - but no client that exports directly to it yet.

Re: OpenTelemetry for Go: Measuring overhead costs

#17

Post by phillipcarter »

orochimaaru wrote: post #13
Thaxll wrote: post #4

Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.

Metrics are usually minimal overheard. Traces need to be sampled. Logs need to be sampled at error/critical levels. You also need to be able to dynamically change sampling and log levels. 100% traces are a mess. I didn’t see where he setup sampling.

The post didn't cover sampling, which indeed, significantly reduces overhead in OTel because the spans that aren't sampled aren't ever created, when you head sample at the SDK level. This is more of a concern when doing tail-based sampling only, wherein you will want to trace each request and offload to a sidecar so that export concerns are handled outside your app. And then it routes to a sampler elsewhere in your infrastructure.

FWIW at my former employer we had some fairly loose guidelines for folks around sampling: https://docs.honeycomb.io/manage-data-volume/sample/guidelin...

There's outliers, but the general idea is that there's also a high cost to implementing sampling (especially for nontrivial stuff), and if your volume isn't terribly high then you'll probably eat a lot more in time than paying for the extra data you may not necessarily need.

Re: OpenTelemetry for Go: Measuring overhead costs

#18

Post by mamidon »

jeffbee wrote: post #8

I feel like this is a lesson that unfortunately did not escape Google, even though a lot of these open systems came from Google or ex-Googlers. The overhead of tracing, logs, and metrics needs to be ultra-low. But the (mis)feature whereby a trace span can be sampled post hoc means that you cannot have a nil tracer that does nothing on unsampled traces, because it could become sampled later. And the idea that if a met…

How would you handle the case where you want to trace 100% of errors? Presumably you don't know a trace is an error until after you've executed the thing and paid the price.

Re: OpenTelemetry for Go: Measuring overhead costs

#19

Post by jhoechtl »

Thaxll wrote: post #4

Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.

I am relatively new to the topic. In the sample code of the OP there is no logging right? It's metrics and traces but no logging.

How is logging in OTel?

Re: OpenTelemetry for Go: Measuring overhead costs

#20

Post by phillipcarter »

mamidon wrote: post #18
jeffbee wrote: post #8

I feel like this is a lesson that unfortunately did not escape Google, even though a lot of these open systems came from Google or ex-Googlers. The overhead of tracing, logs, and metrics needs to be ultra-low. But the (mis)feature whereby a trace span can be sampled post hoc means that you cannot have a nil tracer that does nothing on unsampled traces, because it could become sampled later. And the idea that if a met…

How would you handle the case where you want to trace 100% of errors? Presumably you don't know a trace is an error until after you've executed the thing and paid the price.

This is correct. It's a seemingly simple desire -- "always capture whenever there's a request with an error!" -- but the overhead needed to set that up gets complex. And then you start heading down the path of "well THESE business conditions are more important than THOSE business conditions!" and before you know it, you've got a nice little tower of sampling cards assembled. It's still worth it, just a hefty tax at times, and often the right solution is to just pay for more compute and data so that your engineers are spending less time on these meta-level concerns.
Post reply on HN