Live data from Hacker News

Migrating to OpenTelemetry

airplane.dev

31–40 of 76 posts

Re: Migrating to OpenTelemetry

#31

> The data collected from these streams is sent to several vendors including Datadog (for application logs and metrics), Honeycomb (for traces), and Google Cloud Logging (for infrastructure logs). It sounds like they were in a place that a lot of companies are in where they don't have a single pane of glass for observability. One of if not the main benefit I've gotten out of Datadog is having everything in Datadog so…

I feel we hold up single-observability-solution as the Holy Grail, and I can see the argument for it- one place to understand the health of your services.

But I've also been in terrible vendor lock-in situations, being bent over the barrel because switching to a better solution is so damn expensive.

At least now with OTel you have an open standard that allows you to switch easier, but even then I'd rather have 2 solutions that meet my exact observability requirements than a single solution that does everything OKish.

Re: Migrating to OpenTelemetry

#32
post #30
post #28

Earlier quoted context omitted.

I've spent a small amount of time in datadog, lots in grafana, and somewhere in between in honeycomb. Out applications are designed to emit traces, and comparing honeycomb with tracing to a traditional app with metrics and logs, I would choose tracing every time. It annoys me that logs are overlooked in honeycomb, (and metrics are... fine). But, given the choice between a single pane of glass in grafana or having to…

Agreed - honeycomb has been a boon, however some improvements to metric displays and the ability to set the default "board" used in the home page would be very welcome. Also would be pretty happy if there was a way to drop events on the honeycomb side for a way to dynamically filter - e.g. "don't even bother storing this trace if it has a http.status_code Hopefully someone that works there is reading this.

It sounds like you should look into their tail-sampling Refinery tool https://docs.honeycomb.io/manage-data-volume/refinery/

Re: Migrating to OpenTelemetry

#34
I really really want to use OTel for a small project but have always had a really tough time finding a path that is cheap or free for a personal project.

In theory you can send telemetry data with OTel to Cloud Watch, but I've struggle to connect the dots with the front end application (e.g. React/Next.js).

Re: Migrating to OpenTelemetry

#35

What is the "first principles" argument that observability decomposes into logs, metrics, and tracing? I see this dogma accepted everywhere, but I'm inquisitive about it

First you had logs. Everyone uses logs because it's easy. Logs are great, but suddenly you're spending a crapton of time or money maintaining terabytes or petabytes of storage and ingest of logs. And even worse, in some cases for these logs, you don't actually care about 99% of the log line and simply want a single number, such as CPU utilization or the value of the shopping cart or latency.

So, someone says, "let's make something smaller and more portable than logs. We need to track numerical data over time more easily, so that we can see pretty charts of when these values are outside of where they should be." This ends up being metrics and a time-series database (TSDB), built to handle not arbitrary lines of text but instead meant to parse out metadata and append numerical data to existing time-series based on that metadata.

Between metrics and logs, you end up with a good idea of what's going on with your infrastructure, but logs are still too verbose to understand what's happening with your applications past a certain point. If you have an application crashing repeatedly, or if you've got applications running slowly, metrics and logs can't really help you there. So companies built out Application Performance Monitoring, meant to tap directly into the processes running on the box and spit out all sorts of interesting runtime metrics and events about not just the applications, but the specific methods and calls those applications are utilizing within their stack/code.

Initially, this works great if you're running these APM tools on a single box within monolithic stacks, but as the world moved toward Cloud Service Providers and containerized/ephemeral infrastructure, APM stopped being as effective. When a transaction starts to go through multiple machines and microservices, APM deployed on those boxes individually can't give you the context of how these disparate calls relate to a holistic transaction.

So someone says, "hey, what if we include transaction IDs in these service calls, so that we can post-hoc stitch together these individual transaction lines into a whole transaction, end-to-end?" Which is how you end up with the concept of spans and traces, taking what worked well with Application Performance Monitoring and generalizing that out into the modern microservices architectures that are more common today.

Re: Migrating to OpenTelemetry

#36

> Moreover, we encountered some rough edges in the metrics-related functionality of the Go SDK referenced above. Ultimately, we had to write a conversion layer on top of the OTel metrics API that allowed for simple, Prometheus-like counters, gauges, and histograms. Have encountered this a lot from teams attempting to use the metrics SDK. Are you open to comment on specifics here and also what kind of shim you had to…

Sure, happy to provide more specifics! Our main issue was the lack of a synchronous gauge. The officially supported asynchronous API of registering a callback function to report a gauge metric is very different from how we were doing things before, and would have required lots of refactoring of our code. Instead, we wrote a wrapper that exposes a synchronous-like API: https://gist.github.com/yolken-airplane/027867b75…

the official SDKs will only support an api once there's a spec that allows it.

for const attributes, generally these should be defined at the resource / provider level: https://pkg.go.dev/go.opentelemetry.io/otel/sdk/metric#WithR...

Re: Migrating to OpenTelemetry

#37
post #28

> The data collected from these streams is sent to several vendors including Datadog (for application logs and metrics), Honeycomb (for traces), and Google Cloud Logging (for infrastructure logs). It sounds like they were in a place that a lot of companies are in where they don't have a single pane of glass for observability. One of if not the main benefit I've gotten out of Datadog is having everything in Datadog so…

I've spent a small amount of time in datadog, lots in grafana, and somewhere in between in honeycomb. Out applications are designed to emit traces, and comparing honeycomb with tracing to a traditional app with metrics and logs, I would choose tracing every time. It annoys me that logs are overlooked in honeycomb, (and metrics are... fine). But, given the choice between a single pane of glass in grafana or having to…

Have you tried the traces in grafana/tempo yet? https://grafana.com/docs/grafana/latest/panels-visualization...

It seems to miss some aggregation stuff, but also it's improving every time I check. I wonder if anyone's used it in anger yet and how far is it from replacing datadog or honeycomb.

Re: Migrating to OpenTelemetry

#38

I really really want to use OTel for a small project but have always had a really tough time finding a path that is cheap or free for a personal project. In theory you can send telemetry data with OTel to Cloud Watch, but I've struggle to connect the dots with the front end application (e.g. React/Next.js).

grafana cloud, honeycomb, etc have free tiers, though you'll have to watch how much data you send them. or you can self host something like signoz or the elastic stack. frontend will typically go to an instance of opentelemetry collector to filter/convert to the protocol for the storage backend.

Re: Migrating to OpenTelemetry

#39
post #27

I would love to save a few hundred thousands a year by running Otel collector over Datadog agents, just on the cost-per-host alone. Unfortunately that would also mean giving up Datatog APM and NPM, as far as I can tell, which have been really valuable. Going back to just metrics and traces would feel like quite the step backwards and be a hard sell.

you can submit opentelemetry traces to datadog which should be the equivalent of apm/npm, though maybe with a less polished integration.

Re: Migrating to OpenTelemetry

#40
post #28

Earlier quoted context omitted.

I've spent a small amount of time in datadog, lots in grafana, and somewhere in between in honeycomb. Out applications are designed to emit traces, and comparing honeycomb with tracing to a traditional app with metrics and logs, I would choose tracing every time. It annoys me that logs are overlooked in honeycomb, (and metrics are... fine). But, given the choice between a single pane of glass in grafana or having to…

Have you tried the traces in grafana/tempo yet? https://grafana.com/docs/grafana/latest/panels-visualization... It seems to miss some aggregation stuff, but also it's improving every time I check. I wonder if anyone's used it in anger yet and how far is it from replacing datadog or honeycomb.

tempo still feels very much: look at a trace that you found from elsewhere (like logs).

with so much information in traces and the pure volume, the aggregation really is the key to actionable info out of a tracing setup if it's going to be the primary entry point.

Post reply on HN