Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracing: Structured logging, but better

andydote.co.uk

Post reply on HN
Print view View original

31–40 of 130 posts

Re: Tracing: Structured logging, but better

#31

Post by alexisread »

zoogeny wrote: post #23

One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…

Any of the Clickhouse-based Otel stores can dump the traces to s3 for long-term storage, and can be self-hosted. I know the following use CH: https://uptrace.dev/ https://signoz.io/ https://github.com/hyperdxio/hyperdx

Re: Tracing: Structured logging, but better

#32

Post by thinkharderdev »

lambda_garden wrote: post #27

Couldn't this be injected into the runtime so that no code changes are required? Perhaps really performance critical stuff could have a "notrace" annotation.

Sure, and a lot of tools will do this in one way or another. Either instrument code directly or provide annotations/macros to trace a specific method (something like tokio-tracing in the Rust ecosystem).

However, tracing literally every method call would probably be prohibitively expensive so typically you have either:

1. Instrumentation with "understands" common frameworks/libraries and knows what to instrument (eg request handlers in web frameworks)

2. Full opt-in. They make it easy to add a trace for a method invocation with a simple annotation but nothing gets instrumented by default

Re: Tracing: Structured logging, but better

#33

Post by hosh »

aero142 wrote: post #29
hosh wrote: post #19

That’s weird. I use both logging and tracing where I can. And metrics. While there are better tools for alerting, metrics, or aggregations, it helps a lot in debugging and troubleshooting.

I think the author's point is that tracing is a better implementation of both logs and metrics, and I think it's a valid point. * metrics are pre-aggregated into timeseries data, which makes cardinality expensive. You could also aggregate a value from a trace statement. * Logs are hand crafted and unique, and are usually improved by adding structured attributes. Structured attributes are better as traces because you…

There are things you can do with metrics and logging that you cannot do with traces. These usually fall outside of debugging application performance and bottlenecks. So I think what the author says is true if you are only thinking about application, and not for gaining a holistic understanding of the entire system, including infrastructure.

Probably the biggest tradeoff with traces is that, in practice, you are not retaining 100% of all traces. In order to keep accurate statistics, it generally gets ingested as metrics before sampling. The other is that traces are not stored in such a way where you are looking at what is happening at a point-in-time -- which is what logging does well. If I want to ensure I have execution context for logging, I make the effort to add trace and span ids so that traces and logging can be correlated.

To be fair, I live in the devops world more often than not, and my colleagues on the dev teams rarely have to venture outside of traces.

I don't mind the points this author is making. My main criticism is that it is scoped to the world of applications -- which is fine -- but then taken as universal for all of software engineering.

Re: Tracing: Structured logging, but better

#34

Post by thinkharderdev »

zoogeny wrote: post #23

One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…

> I mean, maybe you end up integrating splunk or perhaps self-hosting kibana

I think this is the issue. Both Splunk and OpenSearch (even self-hosted OpenSearch) get really pricy as well especially with large volumes of log data. Cloudwatch can also get ludicrously expensive. They charge something like $0.50 per GB (!) and another $0.03 per GB to store. I've seen situations at a previous employer where someone accidentally deployed a lambda function with debug logging and ran up a few thousand $$ in Cloudwatch bills overnight.

You should look at Coralogix (disclaimer: I work there). We've built a platform that allows you to store your observability data in S3 and query it through our infrastructure. It can be dramatically more cost-effective than other providers in this space.

Re: Tracing: Structured logging, but better

#35

Post by jlokier »

ducharmdev wrote: post #14

Minor nitpick, but I wish this post started with defining what we mean by logging vs tracing, since some people use these interchangeably. The reader instead has to infer this from the criticisms of logging.

I agree. I'm working with code that uses 'verbose "message"' for level 1 verbosity logs and 'trace "message"' for level 2 verbosity. Makes sense in its world, but it's not the same meaning as how cloud-devops-observability culture uses those words.

Re: Tracing: Structured logging, but better

#36

Post by austinsharp »

lambda_garden wrote: post #27

Couldn't this be injected into the runtime so that no code changes are required? Perhaps really performance critical stuff could have a "notrace" annotation.

Yes, OTel has autoinstrumentation libraries for some language that can pick up a fair amount by default. Though it's unlikely that that would ever be sufficient, it's a nice start.

For Java: https://opentelemetry.io/docs/instrumentation/java/automatic...

Re: Tracing: Structured logging, but better

#37

Post by imiric »

lambda_garden wrote: post #27

Couldn't this be injected into the runtime so that no code changes are required? Perhaps really performance critical stuff could have a "notrace" annotation.

There are several projects that leverage eBPF for automatic instrumentation[1].

How accurate and useful these are vs. doing this manually will depend on the use case, but I reckon the automatic approach gets you most of the way there, and you can add the missing traces yourself, so if nothing else it saves a lot of work.

[1]: https://ebpf.io/applications/

Re: Tracing: Structured logging, but better

#39

Post by hosh »

zoogeny wrote: post #23

One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…

I have made use of tracing, metrics, and logging all together and find each of them have its own place, as well as synergies of being able to work with all three together.

Cost is a real issue, and not just in terms of how much the vendor costs you. When tracing becomes a noticeable fraction of CPU or memory usage relative to the application, it's time to rethink doing 100% sampling. In practice, if you are sampling thousands of requests per second, you're very unlikely to actually look through each one of those thousands (thousands of req/s may not be a lot for some sites, but it is already exceeding human-scale without tooling). In order to keep accurate, useful statistics with sampling, you end up using metrics to store trace metrics prior to sampling.

Re: Tracing: Structured logging, but better

#40

Post by alkonaut »

I like a log to read like a book if it’s the result of a task taking a finite time, such as for example an installation, a compilation, a loading of a browser page or similar. Users are going to look into it for clues about what happened and they a) aren’t always related to those who wrote the tools b) don’t have access to the source code or any special log analytics/querying tools.

That’s when you want a log and that’s what the big traditional log frameworks were designed to handle.

A web backend/service is basically the opposite. End users don’t have access to the log, those who analyze it can cross reference with system internals like source code or db state and the log is basically infinite. In that situation a structured log and querying obviously wins.

It’s honestly not even clear that these systems are that closely related.

Post reply on HN