Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging sucks

loggingsucks.com

Post reply on HN
Print view View original

81–90 of 232 posts

Re: Logging sucks

#81

Post by dexwiz »

ivan_gammel wrote: post #68
spike021 wrote: post #62

Earlier quoted context omitted.

If your codebase has the concept of a request ID, you could also feasibly use that to trace what a user has been doing with more specificity.

…and the same ID can be displayed to user on HTTP 500 with the support contact, making life of everyone much easier.

I have seen pushback on this kind of behavior because "users don't like error codes" or other such nonsense. UX and Product like to pretend nothing will ever break, and when it does they want some funny little image, not useful output.

A good compromise is to log whenever a user would see the error code, and treat those events with very high priority.

Re: Logging sucks

#83

Post by dexwiz »

xmprt wrote: post #72
mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

On the other hand, investing in better tracing tools unlocks a whole nother level of logging and debugging capabilities that aren't feasible with just request logs. It's kind of like you mentioned with using the user id as a "trace" in your first message but on steroids.

These tools tend to be very expensive in my experience unless you are running your own monitoring cloud. Either you end up sampling traces at low rates to save on costs, or your observability bill is more than your infrastructure bill.

Re: Logging sucks

#84

Post by loglog »

dcminter wrote: post #28

I've generally found that structured logs that include a correlation ID make it quite easy to narrow down the general area or exact cause of problems. Usually (in enterprise orgs) via Splunk or Datadog. Where I've had problems it's usually been one of: There wasn't anything logged in the error block. A comment saying "never happens" is often discovered later :) Too much was logged and someone mandated dialing the log…

Java has a solution for the thread problem: Scoped Values [0]. If only the logging+tracing libraries would start using it...

[0] https://openjdk.org/jeps/506

Re: Logging sucks

#85

Post by theodpHN »

Just out of curiosity, how have you seen risk/compliance, regulatory, and audit departments at organizations deal with the disconnect between security and privacy for something like mainframe logging (e.g., JES2, JES3), which is typically inherently governed, and modern distributed logging, which is typically inherently permissive? Both are vastly different approaches, but each is somehow considered 'compliant.' Btw, employees at a company I was at were once investigated for insider trading simply because it was discovered the company used pooled logs that were accessible by production support programmers (the company decided to override the default mainframe security), which was deemed a possible source of insider trading information that could be tapped into by those who had log access (programmers were eventually cleared if it was discovered their small personal trades were immaterial and just coincidental with the company's trading, but the investigation led to uncomfortable confrontations for some!).

Re: Logging sucks

#86

Post by jdpage »

nightpool wrote: post #74
jdpage wrote: post #9

Tangential, but I wonder if the given example might be straying a step too far? Normally we want to keep sensitive data out of logs, but the example includes a user.lifetime_value_cents field. I'd want to have a chat with the rest of the business before sticking something like that in logs.

In some companies, this type of information is often very important and very easily available to everyone at all levels of the business to help prioritize and understand customer value. I would not consider it "sensitive" in the same way that e.g. PII would be.

Good to know! At previous jobs, that information wasn't available to me (and it didn't matter because the customer bases were small enough that every customer was top priority), so I assumed it was considered more sensitive than it perhaps is.

Re: Logging sucks

#87

Post by shireboy »

Kinda get what he’s saying: provide more metadata with structured logging as opposed to lots of string only logs. Ok, modern logging frameworks steer you towards that anyway. But as a counterpoint: often it can be hard to safely enrich logging like that. In the example they include subscription age, user info, etc. More than once I’ve seen logging code lookup metadata or assume it existed, only to cause perf issues or outright errors as expected data didn’t exist. Similar with sampling, it can be frustrating when the thing you need gets sampled out. In the end “it depends” on scenario, but I still find myself not logging enough or else logging too much

Re: Logging sucks

#88

Post by rjbwork »

heinrichhartman wrote: post #4

A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept. Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports gen…

Nick Blumhardt for a while longer than that as "structured logging". Seq and Serilog as enabling software and library in the .net ecosystem.

Re: Logging sucks

#89

Post by Lord_Zero »

Structured Logging is not just JSON. It's the use of templates with context. It solves 90% of what this article complains about if you just log the template along with the variables and the message separately. Along with logging the right stuff. IE `"User {username} created order {orderid}"`
Post reply on HN