Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracing: Structured logging, but better

andydote.co.uk

Post reply on HN
Print view View original

91–100 of 130 posts

Re: Tracing: Structured logging, but better

#91

Post by chii »

layer8 wrote: post #54
beachy wrote: post #49

Earlier quoted context omitted.

> In addition, something that is an error in a subcomponent may only be a warning or even just an info on the level of the superordinate component. Or, keep it simple. - error means someone is alerted urgently to look at the problem - warning means someone should be looking into it eventually, with a view to reclassifying as info/debug or resolving it. IMO many people don't care much about their logs, until the shit…

> - error means someone is alerted urgently to look at the problem The issue is that the code that encounters the problem may not have the knowledge/context to decide whether it warrants alerting. The code higher up that does have the knowledge, on the other hand, often doesn’t have the lower-level information that is useful to have in the log for analyzing the failure. So how do you link the two? When you write modu…

> When you write modular code that minimizes assumptions about its context, that situation is a common occurrence.

so your code isn't modular after all, because the code is _doing_ logging as a side-effect of the actual functionality.

The modularity of your code should mean that the outcome of the functionality is packaged into a bundle of data, and this bundle includes information about errors (or warnings) - aka, a status result.

The caller of this module will inspect this data, and they themselves will decide to log (or, if they are a module of their own, pass the data up again). This goes on, until the data goes into a logging layer - solely responsible for logging perhaps.

Re: Tracing: Structured logging, but better

#92

Post by hardwaresofton »

andrewstuart2 wrote: post #63
hardwaresofton wrote: post #59

I think there's an alternate universe out there where: - we collectively realized that logs, events, traces, metrics, and errors are actually all just logs - we agreed on a single format that encapsulated all that information in a structured manner - we built firehose/stream processing tooling to provide modern o11y creature comforts I can't tell if that universe is better than this one, or worse.

Traces are just distributed "logs" (in the data structure sense; data ordered only by its appearance in something ) where you also pass around the tiniest bit of correlation context between apps. Traces are structured, timestamped, and can be indexed into much more debug-friendly structures like a call tree. But you could just as easily ignore all the data and print them out in streaming sorted order without any corr…

So this is kind of what I was talking about but it's more than that -- if your default is structured logs (simplest example is JSON) then all you have to do is put the data you care about into the log.

So I'm imagining something more like:

   {"level":"info", "otlp": { "trace": { ... }}}

   {"level":"info", "otlp": { "error": { ... }}}

   {"level":"info", "otlp": { "log": { ... }}}

   {"level":"info", "otlp": { "metric": { ... }}}
(standardizing this format would be non-trivial of course, but I could imagine a really minimal standard)

Your downstream collector only needs one API endpoint/ingestion mechanism -- unpacking the actual type of telemetry that came in (and persisting where necessary) can be left to other systems.

Basically I think the systems could have been massively simpler in most UNIX-y environments -- just hook up STDOUT (or scrape it, or syslog or whatever), and you're done -- no allowing ports out for jaeger, dealing with complicated buffering, etc -- just log and forget.

Re: Tracing: Structured logging, but better

#93

Post by hardwaresofton »

phillipcarter wrote: post #61
hardwaresofton wrote: post #59

I think there's an alternate universe out there where: - we collectively realized that logs, events, traces, metrics, and errors are actually all just logs - we agreed on a single format that encapsulated all that information in a structured manner - we built firehose/stream processing tooling to provide modern o11y creature comforts I can't tell if that universe is better than this one, or worse.

That's more or less the model Honeycomb uses. Every signal type is just a structured event. Reality is a bit messier, though. In particular, metrics are the oddball in this world and required a lot of work to make economical.

Ah thanks for noting this, I that's exactly the insight I mean here.

Yeah I think the worst case you basically just exfiltrate metrics out to other subsystems (honestly, you could kind of exfiltrate all of this), but the default is pipe heavily compressed stuff to short and long term storage, and some processors for real time... blah blah blah.

Obviously Honeycomb is actually doing the thing and it's not as easy as it sounds, but it feels like if we had all thought like this earlier we might have skipped making a few protocols (zipkin, jaeger, etc), and focused on just data layout (JSON vs protobuf vs GELF, etc) and figuring out what shapes to expect across tools.

Re: Tracing: Structured logging, but better

#94

Post by hardwaresofton »

dalyons wrote: post #62
hardwaresofton wrote: post #59

I think there's an alternate universe out there where: - we collectively realized that logs, events, traces, metrics, and errors are actually all just logs - we agreed on a single format that encapsulated all that information in a structured manner - we built firehose/stream processing tooling to provide modern o11y creature comforts I can't tell if that universe is better than this one, or worse.

Is that really an alternate universe? That’s the universe that splunk and friends are selling, everything’s a log. It’s really expensive.

Splunk does have margins and I think they're quite high. Same with Datadog (see: all the HN startups that are trying to grab some of that space).

There's a big gap between what it takes for the engineering to work and what all these companies charge.

My point is really more about the engineering time wasted on different protocols and stuff when we could have stuffed everything into minimally structured log lines (and figured out the rest of the insight machinery later). Concretely, that zipkin/jaeger/prometheus protocols and stuff may not have needed to exist, etc.

Re: Tracing: Structured logging, but better

#95

Post by makeavish »

_moog wrote: post #82
phillipcarter wrote: post #26

Earlier quoted context omitted.

FWIW part of the reason you're seeing that is, at least traditionally, APM companies rebranding as Observability companies stuffed trace data into metrics data stores, which becomes prohibitively expensive to query with custom tags/attributes/fields. Newer tools/companies have a different approach that makes cost far more predictable and generally lower. Luckily, some of the larger incumbents are also moving away fro…

> Newer tools/companies have a different approach that makes cost far more predictable and generally lower. What newer tools/companies are in this category? Any that you recommend?

Companies like https://signoz.io/ are Opentelemetry native and have very transparent approach to predictable pricing. You can self host easily as well.

Re: Tracing: Structured logging, but better

#96

Post by BiteCode_dev »

zoogeny wrote: post #23

One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…

You don't have to keep traces for long though.

Log for long term, traces for short debut and analisys is a fine compromise.

Re: Tracing: Structured logging, but better

#97

Post by marcus_holmes »

I'm fundamentally uncomfortable with sending all my data to a third party.

The cool thing about logs is that they're just a text file and don't need to be sent over the internet to someone else. But yes, I've encountered some problems just using text logs and I'd like to solve them.

Is there an OpenTelemetry solution that is capable of being self-hosted (and preferably OS) that anyone recommends?

Re: Tracing: Structured logging, but better

#98

Post by malinens »

marcus_holmes wrote: post #97

I'm fundamentally uncomfortable with sending all my data to a third party. The cool thing about logs is that they're just a text file and don't need to be sent over the internet to someone else. But yes, I've encountered some problems just using text logs and I'd like to solve them. Is there an OpenTelemetry solution that is capable of being self-hosted (and preferably OS) that anyone recommends?

open source tracing tools can be easily locally hosted

Re: Tracing: Structured logging, but better

#99

Post by ec109685 »

dalyons wrote: post #62
hardwaresofton wrote: post #59

I think there's an alternate universe out there where: - we collectively realized that logs, events, traces, metrics, and errors are actually all just logs - we agreed on a single format that encapsulated all that information in a structured manner - we built firehose/stream processing tooling to provide modern o11y creature comforts I can't tell if that universe is better than this one, or worse.

Is that really an alternate universe? That’s the universe that splunk and friends are selling, everything’s a log. It’s really expensive.

Once you have logs, you can index them in a variety of ways to turn them into metrics, traces, etc., but having logs as the fundamental primitive is powerful.

Re: Tracing: Structured logging, but better

#100

Post by ec109685 »

perpil wrote: post #68

I was recently musing about the 2 different types of logs: 1. application logs, emitted multiple times per request and serve as breadcrumbs 2. request logs emitted once per request and include latencies, counters and metadata about the request and response The application logs were useless to me except during development. However the request logs I could run aggregations on which made them far more useful for answeri…

Stripe is big believer in request logs: https://stripe.com/blog/canonical-log-lines
Post reply on HN