Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating to OpenTelemetry

airplane.dev

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Migrating to OpenTelemetry

#51

Post by ankit01-oss »

maccard wrote: post #28
CSMastermind wrote: post #19

> The data collected from these streams is sent to several vendors including Datadog (for application logs and metrics), Honeycomb (for traces), and Google Cloud Logging (for infrastructure logs). It sounds like they were in a place that a lot of companies are in where they don't have a single pane of glass for observability. One of if not the main benefit I've gotten out of Datadog is having everything in Datadog so…

I've spent a small amount of time in datadog, lots in grafana, and somewhere in between in honeycomb. Out applications are designed to emit traces, and comparing honeycomb with tracing to a traditional app with metrics and logs, I would choose tracing every time. It annoys me that logs are overlooked in honeycomb, (and metrics are... fine). But, given the choice between a single pane of glass in grafana or having to…

You can also check out SigNoz - https://github.com/SigNoz/signoz. It has logs, metrics, and traces under a single pane. If you're using otel libraries and otel collector you can do a lot of correlation between your logs and traces. I am a maintainer, and we have seen a lot of our users using signoz to have the ease of having three signals in a single pane.

Re: Migrating to OpenTelemetry

#52

Post by serverlessmom »

maccard wrote: post #28
CSMastermind wrote: post #19

> The data collected from these streams is sent to several vendors including Datadog (for application logs and metrics), Honeycomb (for traces), and Google Cloud Logging (for infrastructure logs). It sounds like they were in a place that a lot of companies are in where they don't have a single pane of glass for observability. One of if not the main benefit I've gotten out of Datadog is having everything in Datadog so…

I've spent a small amount of time in datadog, lots in grafana, and somewhere in between in honeycomb. Out applications are designed to emit traces, and comparing honeycomb with tracing to a traditional app with metrics and logs, I would choose tracing every time. It annoys me that logs are overlooked in honeycomb, (and metrics are... fine). But, given the choice between a single pane of glass in grafana or having to…

I think Honeycomb is perfect for one kind of user, who's entirely concerned with traces and very long retention. For a more general OpenTelemetry-native solution, check out Signoz.

Re: Migrating to OpenTelemetry

#53

Post by Jedd »

pranay01 wrote: post #50
Jedd wrote: post #45

The killer feature of OpenTelemetry for us is brokering (with ETL). Partly this lets us easily re-route & duplicate telemetry, partly it means changes to backend products in the future won't be a big disruption. For metrics we're a mostly telegraf->prometheus->grafana mimir shop - telegraf because its rock solid and feature-rich, prometheus because there's no real competition in that tier, and mimir because of scale…

Yeah, the ability to send to multiple sources is quite powerful and most of this comes from the configurability of Otel Collector [1]. If you are looking for a open source backend for OpenTelemetry, then you can explore SigNoz[2] (I am one of the founders) We have a quite a decent product for APM/tracing leveraging opentelemerty native data format and semantic convention. [1] https://opentelemetry.io/docs/collector/…

Hi Pranay - actually I've had a signoz tab open for about 5 weeks - once I find time I'm meaning to run it up in my lab.

Re: Migrating to OpenTelemetry

#54

Post by pranay01 »

Jedd wrote: post #53
pranay01 wrote: post #50

Earlier quoted context omitted.

Yeah, the ability to send to multiple sources is quite powerful and most of this comes from the configurability of Otel Collector [1]. If you are looking for a open source backend for OpenTelemetry, then you can explore SigNoz[2] (I am one of the founders) We have a quite a decent product for APM/tracing leveraging opentelemerty native data format and semantic convention. [1] https://opentelemetry.io/docs/collector/…

Hi Pranay - actually I've had a signoz tab open for about 5 weeks - once I find time I'm meaning to run it up in my lab.

Awesome! Do reach out to us in our slack community[1] if you have any questions or need any help on setting things up

[1] https://signoz.io/slack

Re: Migrating to OpenTelemetry

#55

Post by nevon »

arccy wrote: post #39
nevon wrote: post #27

I would love to save a few hundred thousands a year by running Otel collector over Datadog agents, just on the cost-per-host alone. Unfortunately that would also mean giving up Datatog APM and NPM, as far as I can tell, which have been really valuable. Going back to just metrics and traces would feel like quite the step backwards and be a hard sell.

you can submit opentelemetry traces to datadog which should be the equivalent of apm/npm, though maybe with a less polished integration.

Just traces are a long way off from APM and NPM. APM gives me the ability to debug memory leaks from continuous heap snapshots, or performance issues through CPU profiling. NPM is almost like having tcpdump running constantly, showing me where there's packet loss or other forms of connectivity issues.

Re: Migrating to OpenTelemetry

#56

Post by shric »

dexterdog wrote: post #26
CSMastermind wrote: post #19

> The data collected from these streams is sent to several vendors including Datadog (for application logs and metrics), Honeycomb (for traces), and Google Cloud Logging (for infrastructure logs). It sounds like they were in a place that a lot of companies are in where they don't have a single pane of glass for observability. One of if not the main benefit I've gotten out of Datadog is having everything in Datadog so…

Depending on your usage it can be prohibitively expensive to use datadog for everything like that. We have it for just our prod env because it's just not worth what it brings to the table to put all of our logs into it.

I once worked out what it would cost to send our company's prod logs to datadog. It was 1.5x our total AWS cost. The company ran entirely on AWS

Re: Migrating to OpenTelemetry

#57

Post by nullify88 »

One thing that's slightly off putting about OpenTelemetry is how resource attributes don't get included as prometheus labels for metrics, instead they are on an info metric which requires a join to enrich the metric you are interested in.

Luckily the prometheus exporters have a switch to enable this behaviour, but there's talk of removing this functionality because it breaks the spec. If you were to use the OpenTelemetry protocol in to something like Mimir, you don't have the option of enabling that behaviour unless you use prometheus remote write.

Our developers aren't a fan of that.

https://opentelemetry.io/docs/specs/otel/compatibility/prome...

Re: Migrating to OpenTelemetry

#58

Post by porker »

nevon wrote: post #55
arccy wrote: post #39

Earlier quoted context omitted.

you can submit opentelemetry traces to datadog which should be the equivalent of apm/npm, though maybe with a less polished integration.

Just traces are a long way off from APM and NPM. APM gives me the ability to debug memory leaks from continuous heap snapshots, or performance issues through CPU profiling. NPM is almost like having tcpdump running constantly, showing me where there's packet loss or other forms of connectivity issues.

Thank you for sharing this, I've had "look at tracing" on my to do list for months and assumed it was identical to APM. It seems it won't be a direct substitute, which helps explain the cost difference.

Re: Migrating to OpenTelemetry

#59

Post by k__ »

marcosdumay wrote: post #22
k__ wrote: post #5

I had the impression, logs and metrics are a pre-observability thing.

Observability is about logs and metrics, and pre-observability (I guess you mean the high-level-only records simpler environments keep) is also about logs and metrics. Anything you register to keep track of your environment has the form of either logs or metrics. The difference is about the contents of such logs and metrics.

When I read Observability Engineering, I got the impression it was about long events and tracing, and metrics and logs were a thing of the past people gave up on since the rise of Microservices.

Re: Migrating to OpenTelemetry

#60

Post by jwestbury »

k__ wrote: post #59
marcosdumay wrote: post #22

Earlier quoted context omitted.

Observability is about logs and metrics, and pre-observability (I guess you mean the high-level-only records simpler environments keep) is also about logs and metrics. Anything you register to keep track of your environment has the form of either logs or metrics. The difference is about the contents of such logs and metrics.

When I read Observability Engineering, I got the impression it was about long events and tracing, and metrics and logs were a thing of the past people gave up on since the rise of Microservices.

> metrics and logs were a thing of the past people gave up on since the rise of Microservices

Definitely not the case, and, in fact, probably the opposite is true. In the era of microservices, metrics are absolutely critical to understand the health of your system. Distributed tracing is also only beneficial if you have the associated logs - so that you can understand what each piece of the system was doing for a single unit of work.

Post reply on HN