Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging sucks

loggingsucks.com

Post reply on HN
Print view View original

41–50 of 232 posts

Re: Logging sucks

#41

Post by tetha »

One thing this is missing: Standardization and probably the ECS' idea of "related" fields.

A common problem in a log aggregation is the question if you query for user.id, user_id, userID, buyer.user.id, buyer.id, buyer_user_id, buyer_id, ... Every log aggregation ends up being plagued by this. You need standard field names there, or it becomes a horrible mess.

And for a centralized aggregation, I like ECS' idea of "related". If you have a buyer and a seller, both with user IDs, you'd have a `related.user.id` with both id's in there. This makes it very simple to say "hey, give me everything related to request X" or "give me everything involving user Y in this time frame" (as long as this is kept up to date, naturally)

Re: Logging sucks

#42

Post by vasco »

heinrichhartman wrote: post #4

A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept. Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports gen…

She has good content but no single person branded the term "observability", what the heck. You can respect someone without making wild claims.

Re: Logging sucks

#43

Post by the_mitsuhiko »

worik wrote: post #33
the_mitsuhiko wrote: post #19

Earlier quoted context omitted.

> If a user request is hitting that many things, in my view, that is a deeply broken architecture. If we want it or not, a lot of modern software looks like that. I am also not a particular fan of building software this way, but it's a reality we're facing. In part it's because quite a few services that people used to build in-house are now outsourced to PaaS solutions. Even basic things such as authentication are mo…

> but it's a reality we're facing. Yes. Most software is bad The incentives between managers and technicians are all wrong Bad software is more profitable, over the time frames managers care about, than good software

The reason we end up with very complex systems I don't think is because of incentives between "managers and technicians". If I were to put my finger to it, I would assume it's the very technicians who argued themselves into a world where increased complexity and more dependencies is seen as a good thing.

Fighting complexity is deeply unpopular.

Re: Logging sucks

#44

Post by dcminter »

the_mitsuhiko wrote: post #25
heinrichhartman wrote: post #4

A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept. Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports gen…

> A post on this topic feels incomplete without a shout-out to Charity Majors I concur. In fact, I strongly recommend anyone who has been working with observability tools or in the industry to read her blog, and the back story that lead to honeycomb. They were the first to recognize the value of this type of observability and have been a huge inspiration for many that came after.

Could you drop a few specific posts here that you think are good for someone (me) who hasn't read her stuff before? Looks like there's a decade of stuff on her blog and I'm not sure I want to start at the very beginning...

Re: Logging sucks

#46

Post by KomoD »

thewisenerd wrote: post #8
OsrsNeedsf2P wrote: post #7

Earlier quoted context omitted.

Because it's an ad

it's an ad, for what? i do not see a product upsell anywhere. if it's an ad for the author themselves, then it's a very good one.

At the end there's a form where you can get a "personalized report", I have a feeling that'll advertise some kind of service, it's usually the case.

Re: Logging sucks

#47

Post by 8n4vidtmkvmk »

mrkeen wrote: post #37

> Your logs are lying to you. Not maliciously. They're just not equipped to tell the truth. The best way to equip logs to tell the truth is to have other parts of the system consume them as their source of truth. Firstly: "what the system does" and "what the logs say" can't be two different things. Secondly: developers can't put less info into the logs than they should, because their feature simply won't work without…

That doesn't sound like a good plan. You're coupling logging with business logic. I don't want to have to think if i change a debug string am i going to break something.

Re: Logging sucks

#48

Post by thevinter »

The presentation is fantastic and I loved the interactive examples!

Too bad that all of this effort is spent arguing something which can be summarised as "add structured tags to your logs"

Generally speaking my biggest gripe with wide logs (and other "innovative" solutions to logging) is that whatever perceived benefit you argue for doesn't justify the increased complexity and loss of readability.

We're throwing away `grep "uid=user-123" application.log` to get what? The shipping method of the user attached to every log? Doesn't feel an improvement to me...

P.S. The checkboxes in the wide event builder don't work for me (brave - android)

Re: Logging sucks

#49

Post by bambax »

> Logging Sucks

But does it? Or is it bad logging, or excessive logging, or unsearchable logs?

A client of mine uses SnapLogic, which is a middleware / ETL that's supposed run pipelines in batch mode to pass data around between systems. It generates an enormous amount of logs that are so difficult to access, search and read that they may as well don't exist.

We're replacing all of that with simple Python scripts that do the same thing and generate normal simple logs with simple errors when something's truly wrong or the data is in the wrong format.

Terse logging is what you want, not an exhaustive (and exhausting) torrent of irrelevant information.

Re: Logging sucks

#50

Post by ttoinou »

tetha wrote: post #41

One thing this is missing: Standardization and probably the ECS' idea of "related" fields. A common problem in a log aggregation is the question if you query for user.id, user_id, userID, buyer.user.id, buyer.id, buyer_user_id, buyer_id, ... Every log aggregation ends up being plagued by this. You need standard field names there, or it becomes a horrible mess. And for a centralized aggregation, I like ECS' idea of "r…

I always wondered why we didnt have some kind of fuzzy english words search regexes/tool, that is robust to keyboard typing mistakes, spelling mistake, synonyms, plural, conjugation etc.
Post reply on HN