Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracing: Structured logging, but better

andydote.co.uk

Post reply on HN
Print view View original

121–130 of 130 posts

Re: Tracing: Structured logging, but better

#121

Post by Too »

marcus_holmes wrote: post #97

I'm fundamentally uncomfortable with sending all my data to a third party. The cool thing about logs is that they're just a text file and don't need to be sent over the internet to someone else. But yes, I've encountered some problems just using text logs and I'd like to solve them. Is there an OpenTelemetry solution that is capable of being self-hosted (and preferably OS) that anyone recommends?

Grafana Tempo or Jaeger all-in-one are both OpenTelemetry compatible and easy host yourself on small scale.

Re: Tracing: Structured logging, but better

#122

Post by Too »

amelius wrote: post #79

This is stuff that a debugger is supposed to do for you, for free. This should not require code at the application level, but it should be implemented at the tooling level.

Are you saying every single variable and function call should be logged every time the code runs? In a dream world sure. While we are at it, let's make it possible to freeze all the state from the production system and let me add breakpoints to rewind in time. In the real world, someone has to make a decision what is noise and what is content.

Unless you are talking about profilers, that measure execution time and memory only, but traces are a lot more than only that.

Annotating the code with logs and traces is a UX activity, not for the end users, but for the ops-team. They don't have knowledge of the internals of the code. Logs should be written in the context of levers that ops have control over.

Take the example from the OP: nr of cache hits. It's something ops can control by configuring the cache size, it is something ops can observe and correlate with request-time and network bandwidth. It would require an immensely sophisticated debugger to make all these correlations automatically.

Re: Tracing: Structured logging, but better

#123

Post by TeMPOraL »

Too wrote: post #120
TeMPOraL wrote: post #118

Earlier quoted context omitted.

Yes, except the problem here is that if the app crashes, you'll lose all the messages in the bundle. That's why people tend to use side-effect logging that persists messages immediately. That, and because it keeps timestamps correct. I suppose this approach would make most sense in event-driven apps where no particular processing takes any meaningful amount of time, so you're constantly revisiting the top-level loop,…

App segfaulting before having chance to log is mostly a thing in the past, unless you are writing c++. Any other language will instead have a top level exception handler. If you were to take hard crashes into account, you would even have to log before each operation instead of after, basically reverting to printf-debugging.

> unless you are writing c++

Guilty as charged.

> If you were to take hard crashes into account, you would even have to log before each operation instead of after

Yes, that's exactly what I see done and do for large enough operations (substeps of those operations only log when they're done).

> basically reverting to printf-debugging

That's what logging is, fundamentally. printf debugging, but with your own printf that has a few more knobs.

Re: Tracing: Structured logging, but better

#124

Post by FridgeSeal »

crabbone wrote: post #117
FridgeSeal wrote: post #71

Earlier quoted context omitted.

I do, as does everyone at my work? Along with basically everyone I’ve ever worked with, ever? Like, I develop cli apps, so like, what else would go to stdout that you suppose will interfere?

Nothing will go to stdout! Nothing is the best thing you can have when it comes to program output. Easiest validation! This is also how all Unix commands work -- they don't write to stdout unless you tell them to. But, if there's nothing extraordinary happening during the program execution -- nothing is written. But why would you write your own logs instead of using something built into your language's library? I bel…

> This is also how all Unix commands work -- they don't write to stdout unless you tell them to

Ok? But as per my other comment, I’m not writing CLI apps, it’s mostly services and I have supporting services which harvest the logs from each containers stdout.

> But why would you write your own logs instead of using something built into your language's library?

I’m not writing my own logging setup? I am using the provided tools?? Every language logging library I’ve ever used writes to stdout?

Structlog in Python, nodejs obvs, all the Rust logging libraries I’ve ever used, I know you can configure Java/scala 3 million different ways (hello yes log4j lol), but all the Spark stuff I’ve written has logged to stdout.

Re: Tracing: Structured logging, but better

#125

Post by jimbokun »

lmm wrote: post #89
rewmie wrote: post #52

Earlier quoted context omitted.

> 3) Switch out log4j for Kafka, which will handle the persistence & multiplexing for you. I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem.

> I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem. If it's data you care about then you put it in Kafka, unless you're big enough to use something like Cassandra or rich enough to pay a cloud provider to make redundant…

OK, but then how do you perform ad hoc queries on everything you logged to Kafka when it's time to debug an issue?

There are plenty of well known, battle tested solutions for solving that problem with old school logging.

Re: Tracing: Structured logging, but better

#126

Post by mrkeen »

jimbokun wrote: post #125
lmm wrote: post #89

Earlier quoted context omitted.

> I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem. If it's data you care about then you put it in Kafka, unless you're big enough to use something like Cassandra or rich enough to pay a cloud provider to make redundant…

OK, but then how do you perform ad hoc queries on everything you logged to Kafka when it's time to debug an issue? There are plenty of well known, battle tested solutions for solving that problem with old school logging.

Do you want to debug what happened to your business entities, or do you want to debug what happened in your logs? Because if they're different things, those are different questions.

> There are plenty of well known, battle tested solutions for solving that problem with old school logging.

And you can run them in parallel (and without interference) by having them ingest from Kafka.

Re: Tracing: Structured logging, but better

#127

Post by koliber »

pondidum wrote: post #111
koliber wrote: post #103

Does this naive approach work for anyone to allow a log to be read like a trace: 1. At the start of a request, generate a globally unique traceId 2. Pass this traceId through the whole call stack. 3. Whenever logging, log the traceId as a parameter Now you have a log with many of the plusses of a trace. The only additional cost to the log is the storage of the traceId on every message. If you want to read a trace, se…

Yes, but going to this effort, why not move to tracing instead? A migration path I could see might be: - replace current logging lib with otel logging (sending to same output) - setup tracing - replace logging with tracing over time (I prefer moving the most painful areas of code first)

One benefit is that you only need to send one string value (traceId) through the whole call stack, instead of passing around a trace object that gets built up. It seems lighter and simpler to add to an existing codebase.

Re: Tracing: Structured logging, but better

#128

Post by jasonjmcghee »

pondidum wrote: post #113
jasonjmcghee wrote: post #3

I really enjoyed the content- it's a great article. Note to author: all but the last code block have a very odd mixture of rather large font sizes (at least on mobile) which vary line to line that make them pretty difficult to read. Also the link to "Observability Driven Development." was a blank slide deck AFAICT

They all look fine in "mobile view" in firefox, and on firefox in android. It's all statically rendered html, and I don't see anything weird in the html either. Do you have a screenshot and some device info so I can look a bit more? Thanks

https://pasteboard.co/xNmrHz0YNmgg.png

Happened in safari and brave.

iOS 16, iPhone 13 Pro

Re: Tracing: Structured logging, but better

#129

Post by lmm »

jimbokun wrote: post #125
lmm wrote: post #89

Earlier quoted context omitted.

> I don't think this is a reasonable statement. There are already a few logging agents that support structured logging without dragging in heavyweight dependencies such as Kafka. Bringing up Kafka sounds like a case of a solution looking for a problem. If it's data you care about then you put it in Kafka, unless you're big enough to use something like Cassandra or rich enough to pay a cloud provider to make redundant…

OK, but then how do you perform ad hoc queries on everything you logged to Kafka when it's time to debug an issue? There are plenty of well known, battle tested solutions for solving that problem with old school logging.

> OK, but then how do you perform ad hoc queries on everything you logged to Kafka when it's time to debug an issue?

Again I'd say treat it like data you care about. Use your best guess at a primary identifier as the record key, depending on your data volume do some indexing/pre-aggregation around other facets that you think you might want to query on (which might include materialising everything in ksqldb, or even in some other datastore), and accept that occasionally you're going to have to do a slow full scan.

> There are plenty of well known, battle tested solutions for solving that problem with old school logging.

Splunk was just bought for $28B because none of those "well known, battle tested solutions" are any good. (Splunk also sucks! It just sucks a little less than the other options).

Re: Tracing: Structured logging, but better

#130

Post by BillinghamJ »

layer8 wrote: post #51
BillinghamJ wrote: post #45

Earlier quoted context omitted.

I tend to think of "warning" as - "something unexpected happened, but it was handled safely" And then "error" as - "things are not okay, a developer is going to need to intervene" And errors then split roughly between "must be fixed sometime", and "must be fixed now/ASAP"

> I tend to think of "warning" as - "something unexpected happened, but it was handled safely" It was handled safely at the level where it occurred, but because it was unusual/unexpected, the underlying cause may cause issues later on or higher up. If one were sure it would 100% not indicate any issue, one wouldn’t need to warn about it.

That would indicate an issue - i.e. something we don't want. Just that it's not something where an engineer needs to go and mop up, and in theory would continue to operate correctly indefinitely. I guess correct as in - safe but not necessarily the most desirable behavior
Post reply on HN