OpenTelemetry for Go: Measuring overhead costs • Hacker News Board

Re: OpenTelemetry for Go: Measuring overhead costs

#41

Post by kiitos » Tue, Jun 17, 2025, 10:53 AM UTC

phillipcarter wrote: post #34

kiitos wrote: post #32

Earlier quoted context omitted.

Traces have a very specific data model, and corresponding limitations, which don't really accommodate log events/messages of arbitrary size. The access model for traces is also fundamentally different vs. that of logs.

There are practical limitations mostly with backend analysis tools. OTel does not define a limit on how large a span is. It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.

> There are practical limitations mostly with backend analysis tools

Not just end-of-line analysis tools, but also initiating SDKs, and system agents, and intermediate middle-boxes -- really anything that needs to parse OTel.

Spec > SDK > Trace > Span limits: https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-lim...

Spec > Common > Attribute limits: https://opentelemetry.io/docs/specs/otel/common/#attribute-l...

I know the spec says the default AttributeValueLengthLimit = infinity, but...

> It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.

...I'd love to learn about any OTel-compatible pipeline/system that supports attribute values of arbitrary size! because I've personally not seen anything that lets you get bigger than O(1MB).

Re: OpenTelemetry for Go: Measuring overhead costs

#42

Post by lquerel » Wed, Jun 18, 2025, 11:28 PM UTC

malkia wrote: post #16

sa46 wrote: post #10

Funny timing—I tried optimizing the Otel Go SDK a few weeks ago ( https://github.com/open-telemetry/opentelemetry-go/issues/67... ). I suspect you could make the tracing SDK 2x faster with some cleverness. The main tricks are: - Use a faster time.Now(). Go does a fair bit of work to convert to the Go epoch. - Use atomics instead of a mutex. I sent a PR, but the reviewer caught correctness issues. Atomics are subtle a…

There is an effort to use arrow format for metrics too - https://github.com/open-telemetry/otel-arrow - but no client that exports directly to it yet.

[dead]

Re: OpenTelemetry for Go: Measuring overhead costs

#43

Post by nikolay_sivko » Thu, Jun 19, 2025, 12:23 PM UTC

phillipcarter wrote: post #17

orochimaaru wrote: post #13

Earlier quoted context omitted.

Metrics are usually minimal overheard. Traces need to be sampled. Logs need to be sampled at error/critical levels. You also need to be able to dynamically change sampling and log levels. 100% traces are a mess. I didn’t see where he setup sampling.

The post didn't cover sampling, which indeed, significantly reduces overhead in OTel because the spans that aren't sampled aren't ever created, when you head sample at the SDK level. This is more of a concern when doing tail-based sampling only, wherein you will want to trace each request and offload to a sidecar so that export concerns are handled outside your app. And then it routes to a sampler elsewhere in your i…

As suggested, I measured the overhead at various sampling rates:

No instrumentation (otel is not initialized): CPU=2.0 cores

SAMPLING 0% (otel initialized): CPU=2.2 cores

SAMPLING 10%: CPU=2.5 cores

SAMPLING 50%: CPU=2.6 cores

SAMPLING 100%: CPU=2.9 cores

Even with 0% sampling, OpenTelemetry still adds overhead due to context propagation, span creation, and instrumentation hooks

Re: OpenTelemetry for Go: Measuring overhead costs

#44

Post by orochimaaru » Fri, Jun 20, 2025, 8:33 PM UTC

nikolay_sivko wrote: post #43

phillipcarter wrote: post #17

Earlier quoted context omitted.

The post didn't cover sampling, which indeed, significantly reduces overhead in OTel because the spans that aren't sampled aren't ever created, when you head sample at the SDK level. This is more of a concern when doing tail-based sampling only, wherein you will want to trace each request and offload to a sidecar so that export concerns are handled outside your app. And then it routes to a sampler elsewhere in your i…

As suggested, I measured the overhead at various sampling rates: No instrumentation (otel is not initialized): CPU=2.0 cores SAMPLING 0% (otel initialized): CPU=2.2 cores SAMPLING 10%: CPU=2.5 cores SAMPLING 50%: CPU=2.6 cores SAMPLING 100%: CPU=2.9 cores Even with 0% sampling, OpenTelemetry still adds overhead due to context propagation, span creation, and instrumentation hooks

Thanks!!

Re: OpenTelemetry for Go: Measuring overhead costs

#45

Post by phillipcarter » Sat, Jun 21, 2025, 6:20 PM UTC

kiitos wrote: post #41

phillipcarter wrote: post #34

Earlier quoted context omitted.

There are practical limitations mostly with backend analysis tools. OTel does not define a limit on how large a span is. It’s quite common in LLM Observability to capture full prompts and LLM responses as attributes on spans, for example.

> There are practical limitations mostly with backend analysis tools Not just end-of-line analysis tools, but also initiating SDKs, and system agents, and intermediate middle-boxes -- really anything that needs to parse OTel. Spec > SDK > Trace > Span limits: https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-lim... Spec > Common > Attribute limits: https://opentelemetry.io/docs/specs/otel/common/#attribute-l..…

Well yeah, there are practical limits imposed by the fact that these have to run on real systems. But in practice, you find that you're limited by your backend observability system because it was designed for a world of many events with narrow data, not fewer events with wider data (so-called "wide events").

OTel and the standard toolkit you get with it doesn't prevent you from doing wide events.

Re: OpenTelemetry for Go: Measuring overhead costs

#46

Post by valyala » Sun, Jun 22, 2025, 4:16 PM UTC

baalimago wrote: post #40

What's the performance drop on Prometheus?

Performance drop for exposing application metrics in Prometheus format is close to zero. These metrics are usually some counters, which are updated atomically in a few nanoseconds. Prometheus scrapes these metrics once per 10-30 seconds, so generating the /metrics response in Prometheus text exposition format doesn't need a lot of CPU, especially if using a library optimized for simplicity and speed like https://github.com/VictoriaMetrics/metrics/

Re: OpenTelemetry for Go: Measuring overhead costs

#47

Post by kiitos » Sun, Jun 22, 2025, 5:14 PM UTC

phillipcarter wrote: post #45

kiitos wrote: post #41

Earlier quoted context omitted.

> There are practical limitations mostly with backend analysis tools Not just end-of-line analysis tools, but also initiating SDKs, and system agents, and intermediate middle-boxes -- really anything that needs to parse OTel. Spec > SDK > Trace > Span limits: https://opentelemetry.io/docs/specs/otel/trace/sdk/#span-lim... Spec > Common > Attribute limits: https://opentelemetry.io/docs/specs/otel/common/#attribute-l..…

Well yeah, there are practical limits imposed by the fact that these have to run on real systems. But in practice, you find that you're limited by your backend observability system because it was designed for a world of many events with narrow data, not fewer events with wider data (so-called "wide events"). OTel and the standard toolkit you get with it doesn't prevent you from doing wide events.

"Wide events" describe a structure/schema for incoming data on the "write path" to a system. That's fine. But that data always needs to be transformed, specialized, for use-case specific "read paths" offered by that same system, in order to be efficient. You can "do wide events" on ingest but you always need to transform them to specific (narrow? idk) events/metrics/summarizations/etc. for the read paths, that's the whole challenge of the space.

Re: OpenTelemetry for Go: Measuring overhead costs

#48

Post by phillipcarter » Mon, Jun 23, 2025, 3:40 PM UTC

jeffbee wrote: post #27

phillipcarter wrote: post #22

Earlier quoted context omitted.

I disagree that it's a bad criterion. The case you describe is what sounds difficult, treating one error as part of normal operations and another as not. That should be considered its own kind of error or other form of response, and sampling decisions could take that into consideration (or not).

Another reason against inflating sampling rates on errors is: for system stability you never want to do more stuff during errors than you would normally do. Doing something more expensive during an error can cause your whole system, or elements of it, to latch into an unplanned operating point where they only have the capacity to do the expensive error path, and all of the traffic is throwing errors because of the re…

I mean, this is why you offload data elsewhere to handle things like sampling and filtering and aggregation.

Re: OpenTelemetry for Go: Measuring overhead costs

#49

Post by phillipcarter » Mon, Jun 23, 2025, 3:42 PM UTC

kiitos wrote: post #47

phillipcarter wrote: post #45

Earlier quoted context omitted.

Well yeah, there are practical limits imposed by the fact that these have to run on real systems. But in practice, you find that you're limited by your backend observability system because it was designed for a world of many events with narrow data, not fewer events with wider data (so-called "wide events"). OTel and the standard toolkit you get with it doesn't prevent you from doing wide events.

"Wide events" describe a structure/schema for incoming data on the "write path" to a system. That's fine. But that data always needs to be transformed, specialized, for use-case specific "read paths" offered by that same system, in order to be efficient. You can "do wide events" on ingest but you always need to transform them to specific (narrow? idk) events/metrics/summarizations/etc. for the read paths, that's the…

You…don’t? This is why tools like ClickHouse and Honeycomb are starting to grow, you just aggregate what you need at query time, and the cost to query is not usually too expensive. The tradeoff is each event has a higher per-unit cost, but this is often the more favorable tradeoff.

Re: OpenTelemetry for Go: Measuring overhead costs

#50

Post by kiitos » Mon, Jun 23, 2025, 8:32 PM UTC

phillipcarter wrote: post #49

kiitos wrote: post #47

Earlier quoted context omitted.

"Wide events" describe a structure/schema for incoming data on the "write path" to a system. That's fine. But that data always needs to be transformed, specialized, for use-case specific "read paths" offered by that same system, in order to be efficient. You can "do wide events" on ingest but you always need to transform them to specific (narrow? idk) events/metrics/summarizations/etc. for the read paths, that's the…

You…don’t? This is why tools like ClickHouse and Honeycomb are starting to grow, you just aggregate what you need at query time, and the cost to query is not usually too expensive. The tradeoff is each event has a higher per-unit cost, but this is often the more favorable tradeoff.

> you just aggregate what you need at query time, and the cost to query is not usually too expensive

The entire challenge of observability systems is rooted in the fact that the volume of input data (wide events) on the write path, is astronomically larger than what can ever be directly evaluated by any user-facing system on the read path. Data transformation and specialization and etc. is the whole ball-game. If you can build something directly on top of raw wide-events, and it works for you, that's cool, but it means that you're operating at trivial and non-representative scale.