31–37 of 37 posts
This layout allows us to quickly merge hundreds of millions of samples into a single profile. The only practical limit is protobuf's 2GB message size cap.
This is cool, OTel is getting somewhere. I've found OTel to still have rough edges and it's not really the one stop shop for telemetry we want it to be at $work yet. In particular, no good (sentry-style) exception capturing yet. They also recently changed a lot of metric names(for good reason it seems), which breaks most dashboards you find on the internet. I have been warned by people that OTel isn't mature yet and…
Relatedly: Has anyone profiled the performance and reliability characteristics of rsyslogd (Linux and FreeBSD distributed syslogger, maybe other platforms too) in its mode where it’s shipping logs to a central node? I’ve configured and used it with relatively small (high single digit nodes, bursts of activity to a million or two requests per minute or so) set-ups but have wondered if there’s a reason it’s not a more…
We're doing this with a few dozen GiBs of logs a day (rsylog -> central rsylog -> elasticsearch). It works reliably, but the config is an absolute nightmare, documentation is a mixed bag and troubleshooting often involves deep dives into the C code. We're planning to migrate to Alloy+Loki.
This is cool, OTel is getting somewhere. I've found OTel to still have rough edges and it's not really the one stop shop for telemetry we want it to be at $work yet. In particular, no good (sentry-style) exception capturing yet. They also recently changed a lot of metric names(for good reason it seems), which breaks most dashboards you find on the internet. I have been warned by people that OTel isn't mature yet and…
At Perforator, we also started from Google's beautiful pprof, but then eliminated all nested repeated fields, converging to https://github.com/yandex/perforator/blob/main/perforator/pr... . Repeated fields in protobufs are really memory & CPU hungry. This layout allows us to quickly merge hundreds of millions of samples into a single profile. The only practical limit is protobuf's 2GB message size cap.
This is cool, OTel is getting somewhere. I've found OTel to still have rough edges and it's not really the one stop shop for telemetry we want it to be at $work yet. In particular, no good (sentry-style) exception capturing yet. They also recently changed a lot of metric names(for good reason it seems), which breaks most dashboards you find on the internet. I have been warned by people that OTel isn't mature yet and…
I think that the "issue" around otel is that instrumentation is easy and free (as both in beer and freedom) but then for the dashboarding part is where there are literally tens of different SaaS solutions, all more or less involved with the OTel development itself, that want you to use their solution (and pay $$$ for it). And even if you can go a loooong way with a self-hosted Grafana + Tempo, even Grafana Labs are p…
I use Grafana because it has value for me both at work and for hobby, but it's becoming more painful to use Grafana for hobby so I agree with your point
Earlier quoted context omitted.
I think that the "issue" around otel is that instrumentation is easy and free (as both in beer and freedom) but then for the dashboarding part is where there are literally tens of different SaaS solutions, all more or less involved with the OTel development itself, that want you to use their solution (and pay $$$ for it). And even if you can go a loooong way with a self-hosted Grafana + Tempo, even Grafana Labs are p…
Yeah. The auto dashboard stuff Grafana cloud is doing nowadays is cool (even if it's just Greg in engineering department writing heuristics), but I can't help but feel pissed that the oss dashboards for otel on Grafanas website aren't even up to date. I use Grafana because it has value for me both at work and for hobby, but it's becoming more painful to use Grafana for hobby so I agree with your point
Indeed. But nowadays you can just work 1 hour with Claude Code and get a pretty slick Grafana dashboard for whatever you need