OpenTelemetry profiles enters public alpha
opentelemetry.io
OpenTelemetry profiles enters public alpha
1–10 of 37 posts
Re: OpenTelemetry profiles enters public alpha
#2> Continuously capturing low-overhead performance profiles in production
It suprises me that anything designed by the OTel community could ever meet 'low-overhead' expectations.
Re: OpenTelemetry profiles enters public alpha
#3> Continuously capturing low-overhead performance profiles in production It suprises me that anything designed by the OTel community could ever meet 'low-overhead' expectations.
Anything to actually add?
Re: OpenTelemetry profiles enters public alpha
#4Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more common solution for distributed logging and tracing (yes it doesn’t solve the UI problem for those, but it does solve collecting your logs)
Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything.
Re: OpenTelemetry profiles enters public alpha
#5> Continuously capturing low-overhead performance profiles in production It suprises me that anything designed by the OTel community could ever meet 'low-overhead' expectations.
OTel Profiling SIG maintainer here: I understand your concern, but we’ve tried our best to make things efficient across the protocol and all
involved components.
Please let us know if you find any issues with what we are shipping right now.
Re: OpenTelemetry profiles enters public alpha
#6> Continuously capturing low-overhead performance profiles in production It suprises me that anything designed by the OTel community could ever meet 'low-overhead' expectations.
The reference implementation of the profiler [1] was originally built by the Optimyze team that Elastic then acquired (and donated to OTEL). That team is very good at what they do. For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled.
Some of the OGs from that team later founded Zymtrace [2] and they're doing the same for profiling what happens inside GPUs now!
[1] https://github.com/open-telemetry/opentelemetry-ebpf-profile...
[2] https://zymtrace.com/article/zero-friction-gpu-profiler/
Re: OpenTelemetry profiles enters public alpha
#7Very excited for this. We've used the Elixir version of this at $WORK a handful of times and have found it exceptionally useful.
Re: OpenTelemetry profiles enters public alpha
#8Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more…
People don’t care about syslog. 98% of my colleagues haven’t heard of it.
Re: OpenTelemetry profiles enters public alpha
#9I wonder how this compares to grafana pyroscope, which is really good for this sort of thing and already quite mature:
Re: OpenTelemetry profiles enters public alpha
#10Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more…
People don’t care about syslog. 98% of my colleagues haven’t heard of it.
You are drawing a global conclusion from a tiny sample!