Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenTelemetry for Go: Measuring overhead costs

coroot.com

Post reply on HN
Print view View original

31–40 of 51 posts

Re: OpenTelemetry for Go: Measuring overhead costs

#31

Post by kiitos »

Thaxll wrote: post #4

Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.

> Tracing every http 200 at 10k req/sec is not something you should be doing

You don't know if a request is HTTP 200 or HTTP 500 until it ends, so you have to at least collect trace data for every request as it executes. You can decide whether or not to emit trace data for a request based on its ultimate response code, but emission is gonna be out-of-band of the request lifecycle, and (in any reasonable implementation) amortized such that you really shouldn't need to care about sampling based on outcome. That is, the cost of collection is >> the cost of emission.

If your tracing system can't handle 100% of your traffic, that's a problem in that system; it's definitely not any kind of universal truth... !

Re: OpenTelemetry for Go: Measuring overhead costs

#32

Post by kiitos »

shanemhansen wrote: post #30
jhoechtl wrote: post #19

Earlier quoted context omitted.

I am relatively new to the topic. In the sample code of the OP there is no logging right? It's metrics and traces but no logging. How is logging in OTel?

To me traces (or maybe more specifically spans) are essentially a structured log with a unique ID and a reference to a parent ID. Very open to have someone explain why I'm wrong or why they should be handled separately.

Traces have a very specific data model, and corresponding limitations, which don't really accommodate log events/messages of arbitrary size. The access model for traces is also fundamentally different vs. that of logs.

Re: OpenTelemetry for Go: Measuring overhead costs

#33

Post by phillipcarter »

jhoechtl wrote: post #19
Thaxll wrote: post #4

Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.

I am relatively new to the topic. In the sample code of the OP there is no logging right? It's metrics and traces but no logging. How is logging in OTel?

Logging in OTel is logging with your logging framework of choice. The SDK just requires you initialize the wrapper and it’ll then wrap your existing logging calls and correlate term with a trace/span in active context, if it exists. There is no separate logging API to learn. Logs are exported in a separate pipeline from traces and metrics.

Implementation for many languages are starting to mature, too.

Re: OpenTelemetry for Go: Measuring overhead costs

#34

Post by phillipcarter »

kiitos wrote: post #32
shanemhansen wrote: post #30

Earlier quoted context omitted.

To me traces (or maybe more specifically spans) are essentially a structured log with a unique ID and a reference to a parent ID. Very open to have someone explain why I'm wrong or why they should be handled separately.

Traces have a very specific data model, and corresponding limitations, which don't really accommodate log events/messages of arbitrary size. The access model for traces is also fundamentally different vs. that of logs.

There are practical limitations mostly with backend analysis tools. OTel does not define a limit on how large a span is. It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.

Re: OpenTelemetry for Go: Measuring overhead costs

#35

Post by hamandcheese »

jeffbee wrote: post #27
phillipcarter wrote: post #22

Earlier quoted context omitted.

I disagree that it's a bad criterion. The case you describe is what sounds difficult, treating one error as part of normal operations and another as not. That should be considered its own kind of error or other form of response, and sampling decisions could take that into consideration (or not).

Another reason against inflating sampling rates on errors is: for system stability you never want to do more stuff during errors than you would normally do. Doing something more expensive during an error can cause your whole system, or elements of it, to latch into an unplanned operating point where they only have the capacity to do the expensive error path, and all of the traffic is throwing errors because of the re…

It can also be expensive as in money. Especially if you are a Datadog customer.

Re: OpenTelemetry for Go: Measuring overhead costs

#36

Post by sa46 »

rastignack wrote: post #28
sa46 wrote: post #10

Funny timing—I tried optimizing the Otel Go SDK a few weeks ago ( https://github.com/open-telemetry/opentelemetry-go/issues/67... ). I suspect you could make the tracing SDK 2x faster with some cleverness. The main tricks are: - Use a faster time.Now(). Go does a fair bit of work to convert to the Go epoch. - Use atomics instead of a mutex. I sent a PR, but the reviewer caught correctness issues. Atomics are subtle a…

Would the sync.Pool trick mentionned here: https://hypermode.com/blog/introducing-ristretto-high-perf-g... help ? It’s lossy but might be a good compromise.

It might be. I've seen the trick pop up a few times:

1. https://puzpuzpuz.dev/thread-local-state-in-go-huh

2. https://victoriametrics.com/blog/go-sync-pool/

It's probably too complex for the Otel SDK, but I might give it a spin in my experimental tracing repo.

Re: OpenTelemetry for Go: Measuring overhead costs

#37

Post by amir_jak »

phillipcarter wrote: post #22
jeffbee wrote: post #21

Earlier quoted context omitted.

I wouldn't. "Trace contains an error" is a hideously bad criterion for sampling. If you have some storage subsystem where you always hedge/race reads to two replicas then cancel the request of the losing replica, then all of your traces will contain an error. It is a genuinely terrible feature. Local logging of error conditions is the way to go. And I mean local, not to a central, indexed log search engine; that's al…

I disagree that it's a bad criterion. The case you describe is what sounds difficult, treating one error as part of normal operations and another as not. That should be considered its own kind of error or other form of response, and sampling decisions could take that into consideration (or not).

You can use the OTel Collector for sampling decisions over tracing, it can also be used for reducing log cost before data is sent to Datadog. There's a whole category of telemetry pipeline now for fully managing that (full disclosure, I work for https://www.sawmills.ai which is a smart telemetry management platform)

Re: OpenTelemetry for Go: Measuring overhead costs

#38

Post by jiggawatts »

A standard trick is to only turn on detailed telemetry from a subset of identical worker VMs or container instances.

Sampling is almost always sufficient for most issues, and when it’s not, you can turn on telemetry on all nodes for selected error levels or critical sections.

Re: OpenTelemetry for Go: Measuring overhead costs

#39

Post by nfrankel »

I have a talk on OpenTelemetry that I regularly present at conferences. After it, I often get the question: "But what's the performance overhead?". In general, I answer by another question: "Is it better to go fast blindfolded or slightly slower with full visibility?". Then I advise the person to do their own performance test in their specific context.

I'm very happy somebody took the time to measure it.

Post reply on HN