Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenTelemetry for Go: Measuring overhead costs

coroot.com

Post reply on HN
Print view View original

21–30 of 51 posts

Re: OpenTelemetry for Go: Measuring overhead costs

#21

Post by jeffbee »

mamidon wrote: post #18
jeffbee wrote: post #8

I feel like this is a lesson that unfortunately did not escape Google, even though a lot of these open systems came from Google or ex-Googlers. The overhead of tracing, logs, and metrics needs to be ultra-low. But the (mis)feature whereby a trace span can be sampled post hoc means that you cannot have a nil tracer that does nothing on unsampled traces, because it could become sampled later. And the idea that if a met…

How would you handle the case where you want to trace 100% of errors? Presumably you don't know a trace is an error until after you've executed the thing and paid the price.

I wouldn't. "Trace contains an error" is a hideously bad criterion for sampling. If you have some storage subsystem where you always hedge/race reads to two replicas then cancel the request of the losing replica, then all of your traces will contain an error. It is a genuinely terrible feature.

Local logging of error conditions is the way to go. And I mean local, not to a central, indexed log search engine; that's also way too expensive.

Re: OpenTelemetry for Go: Measuring overhead costs

#22

Post by phillipcarter »

jeffbee wrote: post #21
mamidon wrote: post #18

Earlier quoted context omitted.

How would you handle the case where you want to trace 100% of errors? Presumably you don't know a trace is an error until after you've executed the thing and paid the price.

I wouldn't. "Trace contains an error" is a hideously bad criterion for sampling. If you have some storage subsystem where you always hedge/race reads to two replicas then cancel the request of the losing replica, then all of your traces will contain an error. It is a genuinely terrible feature. Local logging of error conditions is the way to go. And I mean local, not to a central, indexed log search engine; that's al…

I disagree that it's a bad criterion. The case you describe is what sounds difficult, treating one error as part of normal operations and another as not. That should be considered its own kind of error or other form of response, and sampling decisions could take that into consideration (or not).

Re: OpenTelemetry for Go: Measuring overhead costs

#24

Post by reactordev »

Mmmmmmm, the last 8 months of my life wrapped into a blog post but with an ad on the end. Excellent. Basically the same findings as me, my team, and everyone else in the space.

Not being sarcastic at all, it’s tricky. I like that the article called out eBPF and why you would want to disable it for speed but recommends caution. I kept hearing from executives a “single pane of glass” marketing speak and I kept my mouth shut about how that isn’t feasible across the entire organization. Needless to say, they didn’t like that non-answer and so I was canned. What an engineer cared about is different from organization/business metrics and often the two were confused.

I wrote a lot of great otel receivers though. VMware, Veracode, Hashicorp Vault, GitLab, Jenkins, Jira, and the platforms itself.

Re: OpenTelemetry for Go: Measuring overhead costs

#25

Post by phillipcarter »

reactordev wrote: post #24

Mmmmmmm, the last 8 months of my life wrapped into a blog post but with an ad on the end. Excellent. Basically the same findings as me, my team, and everyone else in the space. Not being sarcastic at all, it’s tricky. I like that the article called out eBPF and why you would want to disable it for speed but recommends caution. I kept hearing from executives a “single pane of glass” marketing speak and I kept my mouth…

> I kept hearing from executives a “single pane of glass” marketing speak

It's really unfortunate that Observability vendors lean into this to reinforce it too. What the execs usually care about is engineering workflows consolidating and allowing teams to all "speak the same language" in terms of data, analysis workflows, visualizations, runbooks, etc.

This goal is admirable, but nearly impossible to achieve because it's the exact same problem as solving "we are aligned organizationally", which no organization ever is.

That doesn't mean progress can't be made, but it's always far more complicated than they would like.

Re: OpenTelemetry for Go: Measuring overhead costs

#26

Post by reactordev »

phillipcarter wrote: post #25
reactordev wrote: post #24

Mmmmmmm, the last 8 months of my life wrapped into a blog post but with an ad on the end. Excellent. Basically the same findings as me, my team, and everyone else in the space. Not being sarcastic at all, it’s tricky. I like that the article called out eBPF and why you would want to disable it for speed but recommends caution. I kept hearing from executives a “single pane of glass” marketing speak and I kept my mouth…

> I kept hearing from executives a “single pane of glass” marketing speak It's really unfortunate that Observability vendors lean into this to reinforce it too. What the execs usually care about is engineering workflows consolidating and allowing teams to all "speak the same language" in terms of data, analysis workflows, visualizations, runbooks, etc. This goal is admirable, but nearly impossible to achieve because…

For sure, it’s the ultimate nirvana. Let me know when an organization gets there. :)

Re: OpenTelemetry for Go: Measuring overhead costs

#27

Post by jeffbee »

phillipcarter wrote: post #22
jeffbee wrote: post #21

Earlier quoted context omitted.

I wouldn't. "Trace contains an error" is a hideously bad criterion for sampling. If you have some storage subsystem where you always hedge/race reads to two replicas then cancel the request of the losing replica, then all of your traces will contain an error. It is a genuinely terrible feature. Local logging of error conditions is the way to go. And I mean local, not to a central, indexed log search engine; that's al…

I disagree that it's a bad criterion. The case you describe is what sounds difficult, treating one error as part of normal operations and another as not. That should be considered its own kind of error or other form of response, and sampling decisions could take that into consideration (or not).

Another reason against inflating sampling rates on errors is: for system stability you never want to do more stuff during errors than you would normally do. Doing something more expensive during an error can cause your whole system, or elements of it, to latch into an unplanned operating point where they only have the capacity to do the expensive error path, and all of the traffic is throwing errors because of the resource starvation.

Re: OpenTelemetry for Go: Measuring overhead costs

#28

Post by rastignack »

sa46 wrote: post #10

Funny timing—I tried optimizing the Otel Go SDK a few weeks ago ( https://github.com/open-telemetry/opentelemetry-go/issues/67... ). I suspect you could make the tracing SDK 2x faster with some cleverness. The main tricks are: - Use a faster time.Now(). Go does a fair bit of work to convert to the Go epoch. - Use atomics instead of a mutex. I sent a PR, but the reviewer caught correctness issues. Atomics are subtle a…

Would the sync.Pool trick mentionned here: https://hypermode.com/blog/introducing-ristretto-high-perf-g... help ? It’s lossy but might be a good compromise.

Re: OpenTelemetry for Go: Measuring overhead costs

#29

Post by kubectl_h »

Thaxll wrote: post #4

Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.

You have to do the tracing anyway if you are going to sample based on criteria that isn't available at the beginning of the trace (like an error that occurs later in the request) and tail sample. You can head sample of course, but that's going to be the most coarse sampling you can do and you can't sample based on anything but the initial conditions of the trace.

What we have started doing is still tracing every unit of work, but deciding at the root span the level of instrumentation fidelity we want for the trace based on the initial conditions. Spans are still generated in the lifecycle of the trace, but we discard them at the processor level (before they are batched and sent to the collector) unless they have errors on them or the trace has been marked as "full fidelity".

Re: OpenTelemetry for Go: Measuring overhead costs

#30

Post by shanemhansen »

jhoechtl wrote: post #19
Thaxll wrote: post #4

Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.

I am relatively new to the topic. In the sample code of the OP there is no logging right? It's metrics and traces but no logging. How is logging in OTel?

To me traces (or maybe more specifically spans) are essentially a structured log with a unique ID and a reference to a parent ID.

Very open to have someone explain why I'm wrong or why they should be handled separately.

Post reply on HN