Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracing: Structured logging, but better

andydote.co.uk

Post reply on HN
Print view View original

11–20 of 130 posts

Re: Tracing: Structured logging, but better

#11

Post by hinkley »

h1fra wrote: post #8

Tracing is much more actionnable but barely usable without a platform. Which makes local programming dependent on third party. Also it requires passing context or have a way to get back the context in every function that requires it, which can be daunting. On my side I have opted to mixed structured/text, a generic message that can be easily understood while glancing over logs, and a data object attached for more det…

Someone got me excited about tracing and I started tweaking our stats API to optionally add tracing. Retrofitted it into a mature app, then immediately discovered that all of the data was being dropped because AWS only likes very tiny traces. Depth or fanout or both break it rather quickly.

And OpenTelemetry has a very questionable implementation. For a nested trace, events fire when the trace closes, meaning that a parent ID is reported before it is seen in the stream. That can’t be good for processing. Would be better to have a leading edge event (also helps with errors throwing and the parent never being reported).

Kind of a bummer. Needs work.

Re: Tracing: Structured logging, but better

#12

Post by discodachshund »

candiddevmike wrote: post #10
waffletower wrote: post #7

Earlier quoted context omitted.

Indeed, the three legs (metrics, logs, traces) of OpenTelemetry's telescope. https://opentelemetry.io

Something missing from OTel IMO is a standard way of linking all three together. It seems like an exercise left to the reader, but I feel like there should be standard metadata for showing a relationship between traces, metrics, and logs. Right now each of these functions is on an island (same with the tooling and storage of the data, but that's another rant).

Isn't that the trace ID? For metrics, it's in the form of exemplars, and for logs it is the log context

Re: Tracing: Structured logging, but better

#13

Post by fnordpiglet »

Tracing is poor at both very long lived traces, at stream processing, and most tracing implementations are too heavy to run in computationally bound tasks beyond at a very coarse level. Logging is nice in that it has no context, no overhead, is generally very cheap to compose and emit, and with including transaction id and done in a structured way gives you most of what tracing does without all the other baggage.

That said for the spaces where tracing works well, it works unreasonably well.

Re: Tracing: Structured logging, but better

#15

Post by pcthrowaway »

hinkley wrote: post #11
h1fra wrote: post #8

Tracing is much more actionnable but barely usable without a platform. Which makes local programming dependent on third party. Also it requires passing context or have a way to get back the context in every function that requires it, which can be daunting. On my side I have opted to mixed structured/text, a generic message that can be easily understood while glancing over logs, and a data object attached for more det…

Someone got me excited about tracing and I started tweaking our stats API to optionally add tracing. Retrofitted it into a mature app, then immediately discovered that all of the data was being dropped because AWS only likes very tiny traces. Depth or fanout or both break it rather quickly. And OpenTelemetry has a very questionable implementation. For a nested trace, events fire when the trace closes, meaning that a…

> OpenTelemetry has a very questionable implementation

The nice thing about OpenTelemetry is that it's a standard. The questionable implementation you're referencing isn't a source of truth. There isn't some canonical "questionable" implementation.

There are many, slightly different, questionable implementations.

Re: Tracing: Structured logging, but better

#16

Post by hinkley »

pcthrowaway wrote: post #15
hinkley wrote: post #11

Earlier quoted context omitted.

Someone got me excited about tracing and I started tweaking our stats API to optionally add tracing. Retrofitted it into a mature app, then immediately discovered that all of the data was being dropped because AWS only likes very tiny traces. Depth or fanout or both break it rather quickly. And OpenTelemetry has a very questionable implementation. For a nested trace, events fire when the trace closes, meaning that a…

> OpenTelemetry has a very questionable implementation The nice thing about OpenTelemetry is that it's a standard. The questionable implementation you're referencing isn't a source of truth. There isn't some canonical "questionable" implementation. There are many, slightly different, questionable implementations.

If the wire protocol has a bug, that’s not something an implementation can fix.

I’m saying the wire protocol is wrong.

Re: Tracing: Structured logging, but better

#17

Post by ryanklee »

ducharmdev wrote: post #14

Minor nitpick, but I wish this post started with defining what we mean by logging vs tracing, since some people use these interchangeably. The reader instead has to infer this from the criticisms of logging.

I've never encountered this confusion anywhere, so I wouldn't ever think to dispel it. Which isn't to say that I disagree with the more general point that defining your terms is good thing.

In any case, the post itself (which is not long) illustrates and marks out many of the differences.

Re: Tracing: Structured logging, but better

#18

Post by candiddevmike »

discodachshund wrote: post #12
candiddevmike wrote: post #10

Earlier quoted context omitted.

Something missing from OTel IMO is a standard way of linking all three together. It seems like an exercise left to the reader, but I feel like there should be standard metadata for showing a relationship between traces, metrics, and logs. Right now each of these functions is on an island (same with the tooling and storage of the data, but that's another rant).

Isn't that the trace ID? For metrics, it's in the form of exemplars, and for logs it is the log context

That might be dependent on the library then, there isn't an official OTel Go logging library yet. Seems you have to add the trace ID exemplars manually too

Re: Tracing: Structured logging, but better

#20

Post by phillipcarter »

fnordpiglet wrote: post #13

Tracing is poor at both very long lived traces, at stream processing, and most tracing implementations are too heavy to run in computationally bound tasks beyond at a very coarse level. Logging is nice in that it has no context, no overhead, is generally very cheap to compose and emit, and with including transaction id and done in a structured way gives you most of what tracing does without all the other baggage. Tha…

Hmmm, for long-lived processes and stream processing we use tracing just fine. What we do is make a cutoff of 60 seconds, which each chunk is its own trace. But our backend queries trace data directly, so we can still analyze the aggregate, long-term behavior and then dig into a particular 60 second chunk if it's problematic.
Post reply on HN