Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenTelemetry profiles enters public alpha

opentelemetry.io

Post reply on HN
Print view View original

11–20 of 37 posts

Re: OpenTelemetry profiles enters public alpha

#11

Post by jbaiter »

genthree wrote: post #4

Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more…

We're doing this with a few dozen GiBs of logs a day (rsylog -> central rsylog -> elasticsearch). It works reliably, but the config is an absolute nightmare, documentation is a mixed bag and troubleshooting often involves deep dives into the C code. We're planning to migrate to Alloy+Loki.

Re: OpenTelemetry profiles enters public alpha

#12

Post by ollien »

SEJeff wrote: post #9

I wonder how this compares to grafana pyroscope, which is really good for this sort of thing and already quite mature: https://grafana.com/oss/pyroscope/ https://github.com/grafana/pyroscope

As far as I'm aware, Pyroscope itself is not a profiler, but a place you can send/query profiles. OpenTelemtry is releasing a profiler, so they don't compare. One can be used with the other.

Re: OpenTelemetry profiles enters public alpha

#13

Post by sciurus »

SEJeff wrote: post #9

I wonder how this compares to grafana pyroscope, which is really good for this sort of thing and already quite mature: https://grafana.com/oss/pyroscope/ https://github.com/grafana/pyroscope

You can send profiles collected by opentelemetry to pyroscope.

https://grafana.com/docs/pyroscope/latest/configure-client/o...

Re: OpenTelemetry profiles enters public alpha

#14

Post by rnrn »

tanelpoder wrote: post #6
secondcoming wrote: post #2

> Continuously capturing low-overhead performance profiles in production It suprises me that anything designed by the OTel community could ever meet 'low-overhead' expectations.

The reference implementation of the profiler [1] was originally built by the Optimyze team that Elastic then acquired (and donated to OTEL). That team is very good at what they do. For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled. Some of the OGs from that team later founded Zymtrace [2] and they're doing the same for profiling what happens in…

> For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled.

This is not an accurate summary of what they developed.

Using .eh_frame to unwind stacks without frame pointers is not novel - it is exactly what it is for and perf has had an implementation doing it since ~2010. The problem is the kernel support for this was repeatedly rejected so the kernel samples kilobytes of stack and then userspace does the unwind

What they developed is an implementation of unwinding from an eBPF program running in the kernel using data from eh_frame.

Re: OpenTelemetry profiles enters public alpha

#15

Post by tanelpoder »

rnrn wrote: post #14
tanelpoder wrote: post #6

Earlier quoted context omitted.

The reference implementation of the profiler [1] was originally built by the Optimyze team that Elastic then acquired (and donated to OTEL). That team is very good at what they do. For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled. Some of the OGs from that team later founded Zymtrace [2] and they're doing the same for profiling what happens in…

> For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled. This is not an accurate summary of what they developed. Using .eh_frame to unwind stacks without frame pointers is not novel - it is exactly what it is for and perf has had an implementation doing it since ~2010. The problem is the kernel support for this was repeatedly rejected so the kernel…

True, I should have been more specific about the context:

Their invention is about pushing down the .eh_frame walking to kernel space, so you don't need to ship large chunks of stack memory to userspace for post-processing. And eBPF code is the executor of that "pushed down" .eh_frame walking.

The GitHub page mentions a patent on this too: https://patents.google.com/patent/US11604718B1/en

Re: OpenTelemetry profiles enters public alpha

#17

Post by ehostunreach »

genthree wrote: post #4

Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more…

Since this is an OTel-related submission, you could also use OTel collectors to collect and forward logs to a central OTel collector instance.

> yes it doesn’t solve the UI problem for those, but it does solve collecting your logs

I work for Netdata and over the last couple months, we've developed an external Netdata plugin that can ingest/index OTel logs [1]. The current implementation stores logs in systemd-compatible journal files and our visualization is effectively the same one someone would get when querying systemd journal logs [2]. i > Like… has anyone done a Jepsen-like stress test on rsyslogd and shared the results? I’ve half-assedly looked before and not been able to find anything.

I've not used rsyslogd specifically, but I don't see how you'd have any issues with the log volume you described.

[1] https://github.com/netdata/netdata/tree/master/src/crates/ne...

[2] https://learn.netdata.cloud/docs/logs/systemd-journal-logs/s...

Re: OpenTelemetry profiles enters public alpha

#18

Post by SEJeff »

ollien wrote: post #12
SEJeff wrote: post #9

I wonder how this compares to grafana pyroscope, which is really good for this sort of thing and already quite mature: https://grafana.com/oss/pyroscope/ https://github.com/grafana/pyroscope

As far as I'm aware, Pyroscope itself is not a profiler, but a place you can send/query profiles. OpenTelemtry is releasing a profiler, so they don't compare. One can be used with the other.

It definitely has profiling client libraries https://grafana.com/docs/pyroscope/latest/configure-client/l...

Re: OpenTelemetry profiles enters public alpha

#19

Post by petterroea »

This is cool, OTel is getting somewhere.

I've found OTel to still have rough edges and it's not really the one stop shop for telemetry we want it to be at $work yet. In particular, no good (sentry-style) exception capturing yet. They also recently changed a lot of metric names(for good reason it seems), which breaks most dashboards you find on the internet.

I have been warned by people that OTel isn't mature yet and I find it to still be true, but it seems the maintainers are trying to do something about this nowadays

Re: OpenTelemetry profiles enters public alpha

#20

Post by [deleted] »

rnrn wrote: post #14
tanelpoder wrote: post #6

Earlier quoted context omitted.

The reference implementation of the profiler [1] was originally built by the Optimyze team that Elastic then acquired (and donated to OTEL). That team is very good at what they do. For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled. Some of the OGs from that team later founded Zymtrace [2] and they're doing the same for profiling what happens in…

> For example, they invented the .eh_frame walking technique to get stack traces from binaries without frame pointers enabled. This is not an accurate summary of what they developed. Using .eh_frame to unwind stacks without frame pointers is not novel - it is exactly what it is for and perf has had an implementation doing it since ~2010. The problem is the kernel support for this was repeatedly rejected so the kernel…

[deleted]
Post reply on HN