Tracing: Structured logging, but better • Hacker News Board

Re: Tracing: Structured logging, but better

#111

Post by pondidum » Thu, Sep 21, 2023, 2:48 PM UTC

koliber wrote: post #103

Does this naive approach work for anyone to allow a log to be read like a trace: 1. At the start of a request, generate a globally unique traceId 2. Pass this traceId through the whole call stack. 3. Whenever logging, log the traceId as a parameter Now you have a log with many of the plusses of a trace. The only additional cost to the log is the storage of the traceId on every message. If you want to read a trace, se…

Yes, but going to this effort, why not move to tracing instead?

A migration path I could see might be:

- replace current logging lib with otel logging (sending to same output) - setup tracing - replace logging with tracing over time (I prefer moving the most painful areas of code first)

Re: Tracing: Structured logging, but better

#112

Post by pondidum » Thu, Sep 21, 2023, 2:54 PM UTC

zoogeny wrote: post #23

One thing about logging and tracing is the inevitable cost (in real money). I love observability probably more than most. And my initial reaction to this article is the obvious: why not both? In fact, I tend to think more in terms of "events" when writing both logs and tracing code. How that event is notified, stored, transmitted, etc. is in some ways divorced from the activity. I don't care if it is going to stdout,…

As other posters have mentioned, the incument companies rebranding to Observability definitely are expensive, because they are charging in the same way as they do for logs and/or metrics: per entry and per unique dimension (metrics especially).

Honeycomb at least charges per event, which in this case means per span - however they don't charge per span attribute, and each span can be pretty large (100kb / 2000 attributes).

I run all my personal services in their free tier, which has plenty of capacity, and that's before I do any sampling.

Re: Tracing: Structured logging, but better

#113

Post by pondidum » Thu, Sep 21, 2023, 3:19 PM UTC

jasonjmcghee wrote: post #3

I really enjoyed the content- it's a great article. Note to author: all but the last code block have a very odd mixture of rather large font sizes (at least on mobile) which vary line to line that make them pretty difficult to read. Also the link to "Observability Driven Development." was a blank slide deck AFAICT

They all look fine in "mobile view" in firefox, and on firefox in android.

It's all statically rendered html, and I don't see anything weird in the html either.

Do you have a screenshot and some device info so I can look a bit more? Thanks

Re: Tracing: Structured logging, but better

#114

Post by rewmie » Thu, Sep 21, 2023, 3:58 PM UTC

Spivak wrote: post #108

rewmie wrote: post #57

Earlier quoted context omitted.

> I think OP meant event sourcing. That is really besides the point. Logging and tracing have always been fundamentally event sourcing, but that never forced anyone ever at all to onboard onto freaking Kafka of all event streaming/messaging platforms. This blend of suggestion sounds an awful lot like resume driven development instead of actually putting together a logging service.

Hard disagree, Kafka is one of the simplest lowest maintenance tools for this with excellent language support and would probably be the first choice for anyone not paying $cloud_vendor for a managed durable queue. The first step in building a reliable logging system is setting up a high write throughout highly available FIFOish durable storage. Once you have that everything else gets a lot easier. * Once the log is c…

> Hard disagree, Kafka is one of the simplest lowest maintenance tools for this (..)

You sound like you've been using an entirely different project named Kafka, because the Kafka everyone uses is renowned among message brokers for its complexity and operational overhead.

Re: Tracing: Structured logging, but better

#115

Post by Spivak » Thu, Sep 21, 2023, 4:48 PM UTC

rewmie wrote: post #114

Spivak wrote: post #108

Earlier quoted context omitted.

Hard disagree, Kafka is one of the simplest lowest maintenance tools for this with excellent language support and would probably be the first choice for anyone not paying $cloud_vendor for a managed durable queue. The first step in building a reliable logging system is setting up a high write throughout highly available FIFOish durable storage. Once you have that everything else gets a lot easier. * Once the log is c…

> Hard disagree, Kafka is one of the simplest lowest maintenance tools for this (..) You sound like you've been using an entirely different project named Kafka, because the Kafka everyone uses is renowned among message brokers for its complexity and operational overhead.

I might be, it's one of the lowest touch services we run. But we aren't doing the "Kafka all the things" model where every single little app is hooked into it for generic message passing but simply logs go in, logs go out, nothing else.

The business logic message passing goes through Rabbit because we wanted out of order processing, priority routing, retry queues, blah blah.

Re: Tracing: Structured logging, but better

#116

Post by pondidum » Thu, Sep 21, 2023, 5:18 PM UTC

lambda_garden wrote: post #27

Couldn't this be injected into the runtime so that no code changes are required? Perhaps really performance critical stuff could have a "notrace" annotation.

Yes, and itel has instrumentation libraries which do this.

However, no automatic instrumentation can do everything for you; it can't know what are all the interesting properties or things to add as attributes. But adding tracing automatically to SQL clients, web frameworks etc is very valuable

Re: Tracing: Structured logging, but better

#117

Post by crabbone » Thu, Sep 21, 2023, 5:40 PM UTC

FridgeSeal wrote: post #71

crabbone wrote: post #25

> The second problem with writing logs to stdout Who on Earth does that? Logs are almost always written to stderr... In part to prevent other problems author is talking about (eg. mixing with the output generated by the application). I don't understand why this has to be either or... If you store the trace output somewhere you get a log... (let's call it "un-annotated" log, since trace won't have the human-readable m…

I do, as does everyone at my work? Along with basically everyone I’ve ever worked with, ever? Like, I develop cli apps, so like, what else would go to stdout that you suppose will interfere?

Nothing will go to stdout! Nothing is the best thing you can have when it comes to program output. Easiest validation! This is also how all Unix commands work -- they don't write to stdout unless you tell them to. But, if there's nothing extraordinary happening during the program execution -- nothing is written.

But why would you write your own logs instead of using something built into your language's library? I believe Python's logging module writes to stderr by default. Go's log package always goes to stderr.

But... today I've learned that console.log() in NodeJS writes to stdout... well, I've lots another tiny bit of faith in humanity.

Re: Tracing: Structured logging, but better

#118

Post by TeMPOraL » Thu, Sep 21, 2023, 6:23 PM UTC

chii wrote: post #91

layer8 wrote: post #54

Earlier quoted context omitted.

> - error means someone is alerted urgently to look at the problem The issue is that the code that encounters the problem may not have the knowledge/context to decide whether it warrants alerting. The code higher up that does have the knowledge, on the other hand, often doesn’t have the lower-level information that is useful to have in the log for analyzing the failure. So how do you link the two? When you write modu…

> When you write modular code that minimizes assumptions about its context, that situation is a common occurrence. so your code isn't modular after all, because the code is _doing_ logging as a side-effect of the actual functionality. The modularity of your code should mean that the outcome of the functionality is packaged into a bundle of data, and this bundle includes information about errors (or warnings) - aka, a…

Yes, except the problem here is that if the app crashes, you'll lose all the messages in the bundle. That's why people tend to use side-effect logging that persists messages immediately. That, and because it keeps timestamps correct.

I suppose this approach would make most sense in event-driven apps where no particular processing takes any meaningful amount of time, so you're constantly revisiting the top-level loop, where the "logging layer" could live. However, most software isn't written this way.

Re: Tracing: Structured logging, but better

#119

Post by lmm » Thu, Sep 21, 2023, 10:01 PM UTC

conradludgate wrote: post #106

lmm wrote: post #90

Earlier quoted context omitted.

I wouldn't call it a "debugger", but plenty of people run an instrumentation agent like New Relic or AppDynamics that records tracing information on their production web services with little or even zero modification to their application code.

> zero modifications Highly depends on the language or framework. I used to use NewRelic APM with Go and it required additional code to instrument. All languages I have used in production require manual instrumentation. For that tracing and logging both work well in my experience.

> I used to use NewRelic APM with Go and it required additional code to instrument.

I'm not surprised, Go's runtime is pretty limited. But for e.g. Java if you're using a well-known/"standard" framework you can just flick the switch and it'll give you a lot of good useful information - additional manual instrumentation usually helps, but the level of instrumentation without it is useful.

Re: Tracing: Structured logging, but better

#120

Post by Too » Fri, Sep 22, 2023, 3:49 AM UTC

TeMPOraL wrote: post #118

chii wrote: post #91

Earlier quoted context omitted.

> When you write modular code that minimizes assumptions about its context, that situation is a common occurrence. so your code isn't modular after all, because the code is _doing_ logging as a side-effect of the actual functionality. The modularity of your code should mean that the outcome of the functionality is packaged into a bundle of data, and this bundle includes information about errors (or warnings) - aka, a…

Yes, except the problem here is that if the app crashes, you'll lose all the messages in the bundle. That's why people tend to use side-effect logging that persists messages immediately. That, and because it keeps timestamps correct. I suppose this approach would make most sense in event-driven apps where no particular processing takes any meaningful amount of time, so you're constantly revisiting the top-level loop,…

App segfaulting before having chance to log is mostly a thing in the past, unless you are writing c++. Any other language will instead have a top level exception handler.

If you were to take hard crashes into account, you would even have to log before each operation instead of after, basically reverting to printf-debugging.