Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging sucks

loggingsucks.com

Post reply on HN
Print view View original

221–230 of 232 posts

Re: Logging sucks

#221

Post by valtism »

mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

Wow, I didn't think this was badly written at all! I certainly don't think it smells like AI. Are you conflating lists with AI written prose?

Re: Logging sucks

#222

Post by valyala »

dexwiz wrote: post #83
xmprt wrote: post #72

Earlier quoted context omitted.

On the other hand, investing in better tracing tools unlocks a whole nother level of logging and debugging capabilities that aren't feasible with just request logs. It's kind of like you mentioned with using the user id as a "trace" in your first message but on steroids.

These tools tend to be very expensive in my experience unless you are running your own monitoring cloud. Either you end up sampling traces at low rates to save on costs, or your observability bill is more than your infrastructure bill.

Try open-source databases specially designed for traces, such as Grafana Tempo or VictoriaTraces. They can handle the data ingestion rate of hundreds of thousands trace spans per second on a regular laptop.

Re: Logging sucks

#223

Post by valyala »

Nice article about the usefulness of wide events! It's pity it doesn't name open-source solutions optimized for wide events such as VictoriaLogs.

Re: Logging sucks

#224

Post by dist1ll »

inkyoto wrote: post #193
dexwiz wrote: post #81

Earlier quoted context omitted.

I have seen pushback on this kind of behavior because "users don't like error codes" or other such nonsense. UX and Product like to pretend nothing will ever break, and when it does they want some funny little image, not useful output. A good compromise is to log whenever a user would see the error code, and treat those events with very high priority.

> I have seen pushback on this kind of behavior because "users don't like error codes" or other such nonsense […] There are two dimensions to it: UX and security. Displaying excessive technical information on an end-user interface will complicate support and likely reveal too much about the internal system design, making it vulnerable to external attacks. The latter is particularly concerning for any design facing th…

Sorry for the OT response, I was curious about this comment[0] you made a while back. How did you measure memory transfer speed?

[0] https://news.ycombinator.com/item?id=38820893

Re: Logging sucks

#225

Post by inkyoto »

dist1ll wrote: post #224
inkyoto wrote: post #193

Earlier quoted context omitted.

> I have seen pushback on this kind of behavior because "users don't like error codes" or other such nonsense […] There are two dimensions to it: UX and security. Displaying excessive technical information on an end-user interface will complicate support and likely reveal too much about the internal system design, making it vulnerable to external attacks. The latter is particularly concerning for any design facing th…

Sorry for the OT response, I was curious about this comment[0] you made a while back. How did you measure memory transfer speed? [0] https://news.ycombinator.com/item?id=38820893

I used «powermetrics» bundled with macOS with «bandwidth» as one of the samplers (--samplers / -s set to «cpu_power,gpu_power,thermal,bandwidth»).

Unfortunately, Apple has taken out the «bandwidth» sampler from «powermetrics», and it is no longer possible to measure the memory bandwidth as easily.

Re: Logging sucks

#227

Post by killme2008 »

This thread overlaps a lot with "Observability 2.0 and the Database for It" (https://news.ycombinator.com/item?id=43789625). The core claim there is: treat logs/spans as structured "wide events", and build a storage/query layer that can handle high-cardinality events so many metrics become derived views rather than pre-modeled upfront. It also argues the hard part isn't "dump it in S3", it’s indexing/queryability + cost control at scale.

In an agentic AI world this pressure gets worse: telemetry becomes more JSON-ish, more high-cardinality (tool names, model/version, prompt/template IDs, step graphs), and more bursty, so pre-modeling every metric up front breaks down faster.

Re: Logging sucks

#228

Post by sahilagarwal »

dexwiz wrote: post #83
xmprt wrote: post #72

Earlier quoted context omitted.

On the other hand, investing in better tracing tools unlocks a whole nother level of logging and debugging capabilities that aren't feasible with just request logs. It's kind of like you mentioned with using the user id as a "trace" in your first message but on steroids.

These tools tend to be very expensive in my experience unless you are running your own monitoring cloud. Either you end up sampling traces at low rates to save on costs, or your observability bill is more than your infrastructure bill.

I like to write them on my own in every company Im in using bash. So I have a local set of bash commands to help me figure out logs and colorize the items I want to.

Takes some time and its a pain in the ass initially, but once I've matured them - work becomes so much more easy. Reduces dependability on other people / teams / access as well.

Edit: Thinking about this, they wont work in other use cases. Im a data engineer so my jobs are mostly sequential.

Re: Logging sucks

#229

Post by weebull »

mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

> - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier.

Depends on the service, but tracking everything a user does may not be an option in terms of data retention laws

Re: Logging sucks

#230

Post by worik »

the_mitsuhiko wrote: post #43
worik wrote: post #33

Earlier quoted context omitted.

> but it's a reality we're facing. Yes. Most software is bad The incentives between managers and technicians are all wrong Bad software is more profitable, over the time frames managers care about, than good software

The reason we end up with very complex systems I don't think is because of incentives between "managers and technicians". If I were to put my finger to it, I would assume it's the very technicians who argued themselves into a world where increased complexity and more dependencies is seen as a good thing. Fighting complexity is deeply unpopular.

> Fighting complexity is deeply unpopular.

Fighting complexity is literally the job of a computer programmer

It is a hard job, and made much harder by the (usual) disconnect between management and us

Post reply on HN