Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating to OpenTelemetry

airplane.dev

Post reply on HN
Print view View original

71–76 of 76 posts

Re: Migrating to OpenTelemetry

#71

Post by valyala »

nullify88 wrote: post #57

One thing that's slightly off putting about OpenTelemetry is how resource attributes don't get included as prometheus labels for metrics, instead they are on an info metric which requires a join to enrich the metric you are interested in. Luckily the prometheus exporters have a switch to enable this behaviour, but there's talk of removing this functionality because it breaks the spec. If you were to use the OpenTelem…

FYI, VictoriaMetrics converts resource attributes to ordinary labels before storing metrics received via OoenTelemetry protocol - https://docs.victoriametrics.com/#sending-data-via-opentelem... . This simplifies filtering and grouping of such metrics during querying. For example, you need to write `my_metric{resource_name="foo"}` instead of `my_metric * on(resource_id) group_left() resource_info{resource_name="foo"}` when filtering by `resource_name`.

Re: Migrating to OpenTelemetry

#72

Post by nullify88 »

valyala wrote: post #71
nullify88 wrote: post #57

One thing that's slightly off putting about OpenTelemetry is how resource attributes don't get included as prometheus labels for metrics, instead they are on an info metric which requires a join to enrich the metric you are interested in. Luckily the prometheus exporters have a switch to enable this behaviour, but there's talk of removing this functionality because it breaks the spec. If you were to use the OpenTelem…

FYI, VictoriaMetrics converts resource attributes to ordinary labels before storing metrics received via OoenTelemetry protocol - https://docs.victoriametrics.com/#sending-data-via-opentelem... . This simplifies filtering and grouping of such metrics during querying. For example, you need to write `my_metric{resource_name="foo"}` instead of `my_metric * on(resource_id) group_left() resource_info{resource_name="foo"}`…

Thanks, that's nice to know VM can accommodate that. A migration will be something we will have consider if Mimir and OpenTelemetry force us to use joins for all our queries.

They maybe trying to address label cardinality but their approach seems like they are throwing the baby out with the bath water. The developer experience suffers as a result because from a dev pov, resource attributes are added to the metric yet this relationship isn't transferred when translated to Prometheus metrics.

Re: Migrating to OpenTelemetry

#73

Post by ronyaurora »

nullify88 wrote: post #57

One thing that's slightly off putting about OpenTelemetry is how resource attributes don't get included as prometheus labels for metrics, instead they are on an info metric which requires a join to enrich the metric you are interested in. Luckily the prometheus exporters have a switch to enable this behaviour, but there's talk of removing this functionality because it breaks the spec. If you were to use the OpenTelem…

If you are using the prometheus exporter, you can use the transform processor to get specific resource attributes into metric labels.

With the advantage that you get only the specific attributes you want, thus avoiding a cardinality explosion.

https://github.com/open-telemetry/opentelemetry-collector-co...

Re: Migrating to OpenTelemetry

#74

Post by nullify88 »

ronyaurora wrote: post #73
nullify88 wrote: post #57

One thing that's slightly off putting about OpenTelemetry is how resource attributes don't get included as prometheus labels for metrics, instead they are on an info metric which requires a join to enrich the metric you are interested in. Luckily the prometheus exporters have a switch to enable this behaviour, but there's talk of removing this functionality because it breaks the spec. If you were to use the OpenTelem…

If you are using the prometheus exporter, you can use the transform processor to get specific resource attributes into metric labels. With the advantage that you get only the specific attributes you want, thus avoiding a cardinality explosion. https://github.com/open-telemetry/opentelemetry-collector-co...

We've migrated away from the prometheus exporter to the prometheus remote write exporter as I'd like a completely "push" based architecture. Ideally I would have liked to be completely otlp but can't for the reasons already explained. So I use promethus remote write instead in to Mimir.

The transform processor could be useful if they ever deprecate the resource_to_telemetry_conversion flag, but its still a pain point because it hinders a developers autonomy, and requires a whitelist of labels to be maintained on the collectors by another team.

Re: Migrating to OpenTelemetry

#75

Post by hagen1778 »

Jedd wrote: post #45

The killer feature of OpenTelemetry for us is brokering (with ETL). Partly this lets us easily re-route & duplicate telemetry, partly it means changes to backend products in the future won't be a big disruption. For metrics we're a mostly telegraf->prometheus->grafana mimir shop - telegraf because its rock solid and feature-rich, prometheus because there's no real competition in that tier, and mimir because of scale…

> mimir because of scale & self-host options

Have you looked at VictoriaMetrics [0] before opting for Mimir?

[0] https://victoriametrics.com/blog/mimir-benchmark/

Re: Migrating to OpenTelemetry

#76

Post by dexterdog »

dabeeeenster wrote: post #42
dexterdog wrote: post #26

Earlier quoted context omitted.

Depending on your usage it can be prohibitively expensive to use datadog for everything like that. We have it for just our prod env because it's just not worth what it brings to the table to put all of our logs into it.

Is prod not 99% of your logs?

Not even close
Post reply on HN