OpenTelemetry for Go: Measuring overhead costs
1–10 of 51 posts
Re: OpenTelemetry for Go: Measuring overhead costs
#2[deleted]
Re: OpenTelemetry for Go: Measuring overhead costs
#3Not on original topic, but:
I definitely prefer having graphs put the unit at least on the axis, if not in the individual axis labels directly.
I.e. instead of having a graph titled "latency, seconds" at the top and then way over on the left have an unlabeled axis with "5m, 10m, 15m, 20m" ticks...
I'd rather have title "latency" and either "seconds" on the left, or, given the confusion between "5m = 5 minutes" or "5m = 5 milli[seconds]", just have it explicitly labeled on each tick: 5ms, 10ms, ...
Way, way less likely to confuse someone when the units are right on the number, instead of floating way over in a different section of the graph
Re: OpenTelemetry for Go: Measuring overhead costs
#4Logging, metrics and traces are not free, especially if you turn them on at every requests.
Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.
Re: OpenTelemetry for Go: Measuring overhead costs
#5Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.
A very small % of startups gets anywhere near that traffic so why give them angst? Most people can just do this without any issues and learn from it and a tiny fraction shouldn't.
Re: OpenTelemetry for Go: Measuring overhead costs
#6Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.
A very small % of startups gets anywhere near that traffic so why give them angst? Most people can just do this without any issues and learn from it and a tiny fraction shouldn't.
10k/s across multiple services is reached quickly even at startup scale.
In my previous company (startup), we’d use Otel everywhere and we definitely needed sampling for cost reasons (1/30 iirc). And that was using a much cheaper provider than Datadog
Re: OpenTelemetry for Go: Measuring overhead costs
#7Logging, metrics and traces are not free, especially if you turn them on at every requests. Tracing every http 200 at 10k req/sec is not something you should be doing, at that rate you should sample 200 ( 1% or so ) and trace all the errors.
A very small % of startups gets anywhere near that traffic so why give them angst? Most people can just do this without any issues and learn from it and a tiny fraction shouldn't.
Having high req/s isn't as big a negative as it once was. Especially if you are using http2 or http3.
Designing APIs which cause a high number of requests and spit out a low amount of data can be quite legitimate. It allows for better scaling and capacity planning vs having single calls that take a large amount of time and return large amounts of data.
In the old http1 days, it was a bad thing because a single connection could only service 1 request at a time. Getting any sort of concurrency or high request rates require many connections (which had a large amount of overhead due to the way tcp functions).
We've moved past that.
Re: OpenTelemetry for Go: Measuring overhead costs
#8I feel like this is a lesson that unfortunately did not escape Google, even though a lot of these open systems came from Google or ex-Googlers. The overhead of tracing, logs, and metrics needs to be ultra-low. But the (mis)feature whereby a trace span can be sampled post hoc means that you cannot have a nil tracer that does nothing on unsampled traces, because it could become sampled later. And the idea that if a metric exists it must be centrally collected is totally preposterous, makes everything far too expensive when all a developer wants is a metric that costs nothing in the steady state but can be collected when needed.
Re: OpenTelemetry for Go: Measuring overhead costs
#9The article never really explains what eBPF is -- AFAIU, it’s a kernel feature that lets you trace syscalls and network events without touching your app code. Low overhead, good for metrics, but not exactly transparent.
It’s the umpteenth OTEL-critical article on the front page of HN this month alone... I have to say I share the sentiment but probably for different reasons. My take is quite the opposite: most value is precisely in the application (code) level so you definetly should instrument... and then focus on Errors over "general observability"[0]
Re: OpenTelemetry for Go: Measuring overhead costs
#10Funny timing—I tried optimizing the Otel Go SDK a few weeks ago (https://github.com/open-telemetry/opentelemetry-go/issues/67...).
I suspect you could make the tracing SDK 2x faster with some cleverness. The main tricks are:
- Use a faster time.Now(). Go does a fair bit of work to convert to the Go epoch.
- Use atomics instead of a mutex. I sent a PR, but the reviewer caught correctness issues. Atomics are subtle and tricky.
- Directly marshal protos instead of reflection with a hand-rolled library or with https://github.com/VictoriaMetrics/easyproto.
The gold standard is how TiDB implemented tracing (https://www.pingcap.com/blog/how-we-trace-a-kv-database-with...). Since Go purposefully (and reasonably) doesn't currently provide a comparable abstraction for thread-local storage, we can't implement similar tricks like special-casing when a trace is modified on a single thread.