Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating to OpenTelemetry

airplane.dev

Post reply on HN
Print view View original

61–70 of 76 posts

Re: Migrating to OpenTelemetry

#61

Post by jon-wood »

At the risk of being downvoted (probably justly) for having a moan, can we please have a moratorium on every blog post needing to have a generally irrelevant picture attached to it? On opening this page I can see 28 words that are actually relevant because almost the entire view is consumed by a huge picture of a graph and the padding around it.

This is endemic now. Doesn't matter what someone is writing about there'll be some pointless stock photo taking up half the page. There'll probably be some more throughout the page. Stop it please.

Re: Migrating to OpenTelemetry

#62

Post by everfrustrated »

rewmie wrote: post #29
CSMastermind wrote: post #19

> The data collected from these streams is sent to several vendors including Datadog (for application logs and metrics), Honeycomb (for traces), and Google Cloud Logging (for infrastructure logs). It sounds like they were in a place that a lot of companies are in where they don't have a single pane of glass for observability. One of if not the main benefit I've gotten out of Datadog is having everything in Datadog so…

> It sounds like they were in a place that a lot of companies are in where they don't have a single pane of glass for observability. One of the biggest features of AWS which is very easy to take for granted and go unnoticed is Amazon CloudWatch. It supports metrics, logging, alarms, metrics from alarms, alarms from alarms, querying historical logs, trigger actions, etc etc etc. and it covers each and every single ser…

In addition, one of the largest limitations of CloudWatch is it doesn't work well with a many-aws-account strategy.

Some part of the value of Datadog etc is having a single pane of glass over many aws accounts.

Re: Migrating to OpenTelemetry

#63

Post by sofixa »

pranay01 wrote: post #48
badloginagain wrote: post #31

Earlier quoted context omitted.

I feel we hold up single-observability-solution as the Holy Grail, and I can see the argument for it- one place to understand the health of your services. But I've also been in terrible vendor lock-in situations, being bent over the barrel because switching to a better solution is so damn expensive. At least now with OTel you have an open standard that allows you to switch easier, but even then I'd rather have 2 solu…

I think that OpenTelemetry will solve this problem of vendor lock in. I am a founder building in this space[1] and we see many of our users switching to opentelemetry as that provides an easy way to switch if needed in future. At SigNoz, we have metrics, traces and logs in a single application which helps you correlate across signals much more easily - and being natively based on opentelemetry makes this correlation…

Switch the backend destination of metrics/traces/logs, but all your dashboards, alerts, and potentially legacy data still need to be migrated. Drastically better than before where instrumentation and agents were custom for each backend, but there's still hurdles.

Re: Migrating to OpenTelemetry

#64

Post by sofixa »

k__ wrote: post #59
marcosdumay wrote: post #22

Earlier quoted context omitted.

Observability is about logs and metrics, and pre-observability (I guess you mean the high-level-only records simpler environments keep) is also about logs and metrics. Anything you register to keep track of your environment has the form of either logs or metrics. The difference is about the contents of such logs and metrics.

When I read Observability Engineering, I got the impression it was about long events and tracing, and metrics and logs were a thing of the past people gave up on since the rise of Microservices.

> Authors Charity Majors, Liz Fong-Jones, and George Miranda from Honeycomb explain what constitutes good observability, show you how to improve upon what you're doing today, and provide practical dos and don'ts for migrating from legacy tooling, such as metrics, monitoring, and log management. You'll also learn the impact observability has on organizational culture (and vice versa).

No wonder, it's either strong bias from people working in a tracing vendor, or outright a sales pitch.

It's totally false though. Each pillar - metrics, logs and traces have their place and serve different purposes. You won't use traces to measure the number of requests hitting your load balancer, or the amount of objects in the async queue, or CPU utilisation, or network latency, or any number of things. Logs can be more rich than traces, and a nice pattern I've used with Grafana is linking the two, and having the option to jump to corresponding log lines from a trace which can describe the different actions performed during that span.

Re: Migrating to OpenTelemetry

#65

Post by maccard »

viraptor wrote: post #37
maccard wrote: post #28

Earlier quoted context omitted.

I've spent a small amount of time in datadog, lots in grafana, and somewhere in between in honeycomb. Out applications are designed to emit traces, and comparing honeycomb with tracing to a traditional app with metrics and logs, I would choose tracing every time. It annoys me that logs are overlooked in honeycomb, (and metrics are... fine). But, given the choice between a single pane of glass in grafana or having to…

Have you tried the traces in grafana/tempo yet? https://grafana.com/docs/grafana/latest/panels-visualization... It seems to miss some aggregation stuff, but also it's improving every time I check. I wonder if anyone's used it in anger yet and how far is it from replacing datadog or honeycomb.

I've not. Honestly, I'm not in the market for tool shopping at the moment, I need another honeycomb-style moment of "this is incredible" to start looking again. I think it would take "Honeycomb, but we handle metric rollups and do logs" right now.

Re: Migrating to OpenTelemetry

#66

Post by phillipcarter »

masterj wrote: post #32
mdtusz wrote: post #30

Earlier quoted context omitted.

Agreed - honeycomb has been a boon, however some improvements to metric displays and the ability to set the default "board" used in the home page would be very welcome. Also would be pretty happy if there was a way to drop events on the honeycomb side for a way to dynamically filter - e.g. "don't even bother storing this trace if it has a http.status_code Hopefully someone that works there is reading this.

It sounds like you should look into their tail-sampling Refinery tool https://docs.honeycomb.io/manage-data-volume/refinery/

Yep, this is the one to use. Refinery handles exactly this scenario (and more).

Re: Migrating to OpenTelemetry

#67

Post by phillipcarter »

sofixa wrote: post #64
k__ wrote: post #59

Earlier quoted context omitted.

When I read Observability Engineering, I got the impression it was about long events and tracing, and metrics and logs were a thing of the past people gave up on since the rise of Microservices.

> Authors Charity Majors, Liz Fong-Jones, and George Miranda from Honeycomb explain what constitutes good observability, show you how to improve upon what you're doing today, and provide practical dos and don'ts for migrating from legacy tooling, such as metrics, monitoring, and log management. You'll also learn the impact observability has on organizational culture (and vice versa). No wonder, it's either strong bia…

You can sorta measure some of this with traces. For example, sampled traces that contain the sampling rate in their metadata let you re-weight counts, thus allowing you to accurately measure "number of requests to x". Similarly, a good sampling of network latency can absolutely be measured by trace data. Metrics will always have their place, though, for reasons you mention - measuring cpu utilization, # of objects in something etc. Logs vs. traces is more nuanced I think. A trace is nothing more than a collection of structured logs. I would wager that nearly all use cases for structured logging could be wholesale replaced by tracing. Security logging and big object logging is an exception, although that's also dependent on your vendor or backend.

Re: Migrating to OpenTelemetry

#68

Post by phillipcarter »

jwestbury wrote: post #60
k__ wrote: post #59

Earlier quoted context omitted.

When I read Observability Engineering, I got the impression it was about long events and tracing, and metrics and logs were a thing of the past people gave up on since the rise of Microservices.

> metrics and logs were a thing of the past people gave up on since the rise of Microservices Definitely not the case, and, in fact, probably the opposite is true. In the era of microservices, metrics are absolutely critical to understand the health of your system. Distributed tracing is also only beneficial if you have the associated logs - so that you can understand what each piece of the system was doing for a sin…

> Distributed tracing is also only beneficial if you have the associated logs - so that you can understand what each piece of the system was doing for a single unit of work.

Ehhh, that's only if you view tracing as "the thing that tells me that service A talks to service B". Spans in a trace are just structured logs. They are your application logging vehicle, especially if you don't have a legacy of good in-app instrumentation via logs.

But even then the worlds are blurring a bit. OTel logs burn in a span and trace ID, and depending on the backend that correlated log may well just be treated as if it's a part of the trace.

Re: Migrating to OpenTelemetry

#69

Post by malkia »

sofixa wrote: post #64
k__ wrote: post #59

Earlier quoted context omitted.

When I read Observability Engineering, I got the impression it was about long events and tracing, and metrics and logs were a thing of the past people gave up on since the rise of Microservices.

> Authors Charity Majors, Liz Fong-Jones, and George Miranda from Honeycomb explain what constitutes good observability, show you how to improve upon what you're doing today, and provide practical dos and don'ts for migrating from legacy tooling, such as metrics, monitoring, and log management. You'll also learn the impact observability has on organizational culture (and vice versa). No wonder, it's either strong bia…

While I was at Google, circa 2015-2016, working on some Ad project, and happened to be on-call our system started doing something wonky, so I think I either called the SRE on the sub-system we were using (spanner? something else - don't remember) to check what's up (as written by our playbook).

They asked me to enable tracing for 30s (we had Chrome extension, that sends some URL common parameter that enables in your web server full tracing (100%) for some short amount of time), and then I did some operations that our internal customers were complaining.

This produced quite a hefty tracing, but only for 30secs - enough for them to trace back where the issue might be coming from. But basically end-to-end from me doing something on the browser, down to our server/backend, downto their systems, etc.

That's how I've learned how important it is - for cases like this, but you can't have 100% ON all the time - not even 1% I think...

Re: Migrating to OpenTelemetry

#70

Post by sofixa »

malkia wrote: post #69
sofixa wrote: post #64

Earlier quoted context omitted.

> Authors Charity Majors, Liz Fong-Jones, and George Miranda from Honeycomb explain what constitutes good observability, show you how to improve upon what you're doing today, and provide practical dos and don'ts for migrating from legacy tooling, such as metrics, monitoring, and log management. You'll also learn the impact observability has on organizational culture (and vice versa). No wonder, it's either strong bia…

While I was at Google, circa 2015-2016, working on some Ad project, and happened to be on-call our system started doing something wonky, so I think I either called the SRE on the sub-system we were using (spanner? something else - don't remember) to check what's up (as written by our playbook). They asked me to enable tracing for 30s (we had Chrome extension, that sends some URL common parameter that enables in your…

Oh yeah, tracing can be extremely useful, precisely because it should be end to end.

As for the numbers, that's why all tracing collectors and receivers support downsampling out of the box. Recording only 1% or 10% of all traces, or 10% of all successful ones and 100% of failures is a good way of making use of tracing without overburdening storage.

Post reply on HN