Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging sucks

loggingsucks.com

Post reply on HN
Print view View original

141–150 of 232 posts

Re: Logging sucks

#141

Post by giancarlostoro »

mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

TIDs are good here too. If you generate it and enforce it across all your services spanning various teams and APIs anyone of any team can grab a TID you provide and you can get the full end to end of one transaction.

Re: Logging sucks

#142

Post by bccdee »

khazhoux wrote: post #123
mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

> That was difficult to read, smelt very AI assisted though the message was worthwhile... It won’t be long before ad computem comments like this are frowned upon.

Why? "This was written badly" is a perfectly normal thing to say; "this was written badly because you didn't put in the effort of writing it yourself" doubly so.

Re: Logging sucks

#143

Post by mkarrmann »

I broadly agree with the article.

The described pattern is standard in Meta. This, along with the infrastructure and tooling to support it, was the single largest "devx quality of life improvement" in my experience moving to big tech.

Re: Logging sucks

#144

Post by dannyfreeman »

thevinter wrote: post #48

The presentation is fantastic and I loved the interactive examples! Too bad that all of this effort is spent arguing something which can be summarised as "add structured tags to your logs" Generally speaking my biggest gripe with wide logs (and other "innovative" solutions to logging) is that whatever perceived benefit you argue for doesn't justify the increased complexity and loss of readability. We're throwing away…

Do you really loose the ability to grep? You can still search for json fragments `grep '"uid": "user-123"' application.log`

If the json logged isn't pretty printed everything should still be on one line. You can also grep with the `--context` flag to get more surrounding lines.

Re: Logging sucks

#145

Post by dpark »

otterley wrote: post #134
dpark wrote: post #127

Earlier quoted context omitted.

Surely you aren’t doing real time indexing, transformation, analytics, etc in the same service that is producing the logs. A catastrophic increase in logging could certainly take down your log processing pipeline but it should not create cascading failures that compromise your service.

Of course not. Worst case should be backpressure, which means processing, indexing, and storage delays. Your service might be fine but your visibility will be reduced.

For sure. Your can definitely tip over your logging pipeline and impact visibility.

I just wanted to make sure we weren’t still talking about “causing a cascading outage due to increased log volumes” as was mentioned above, which would indicate a significant architectural issue.

Re: Logging sucks

#146

Post by alwa »

khazhoux wrote: post #123
mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

> That was difficult to read, smelt very AI assisted though the message was worthwhile... It won’t be long before ad computem comments like this are frowned upon.

I read it as a more-or-less kind comment: “even though you’ll notice that they let an AI make the writing terrible, the underlying point is good enough to be worth struggling through that and discussing”

Re: Logging sucks

#147

Post by the_mitsuhiko »

Aurornis wrote: post #137
the_mitsuhiko wrote: post #25

Earlier quoted context omitted.

> A post on this topic feels incomplete without a shout-out to Charity Majors I concur. In fact, I strongly recommend anyone who has been working with observability tools or in the industry to read her blog, and the back story that lead to honeycomb. They were the first to recognize the value of this type of observability and have been a huge inspiration for many that came after.

> They were the first to recognize the value of this type of observability With all due respect to her great writing, I think there’s a mix of revisionist history blended with PR claims going on in this thread. The blog has some good reading, but let’s not get ahead of ourselves in rewriting history around this one person/company.

> I think there’s a mix of revisionist history blended with PR claims going on in this thread.

I can only speak for myself. I worked for a company that is somewhere in the observability space (Sentry) and Charity was a person I looked up to my entire time working on Sentry. Both for how she ran the company, for the design they picked and for the approaches they took. There might be others that have worked on wide events (afterall, Honeycomb is famously inspired by Facebook's scuba), she is for sure the voice that made it popular.

Re: Logging sucks

#148

Post by 0xbadcafebee »

bccdee wrote: post #142
khazhoux wrote: post #123

Earlier quoted context omitted.

> That was difficult to read, smelt very AI assisted though the message was worthwhile... It won’t be long before ad computem comments like this are frowned upon.

Why? "This was written badly" is a perfectly normal thing to say; "this was written badly because you didn't put in the effort of writing it yourself" doubly so.

Say they used AI to write it, it came out bad, and they published it anyway. They had the opportunity to "make it better" before publishing, but didn't. The only conclusion for this is, they just aren't good at writing. So whether AI is used or not, it'll suck either way. So there's no need to complain about the AI.

It's like complaining that somebody typed a crappy letter rather than hand-wrote it. Either way the letter's gonna suck, so why complain that it was typed?

Re: Logging sucks

#149

Post by UltraSane »

preisschild wrote: post #122
UltraSane wrote: post #80

Splunk is expensive but it makes searching logs so much faster and more effective. I think of it as SQL for unstructured data.

loki works great too and is FOSS

We really need an open-source implementation of the Splunk Query Language. The query language is what lets you actually find the few dozen relevant lines out of the billions of lines logged.

Re: Logging sucks

#150

Post by nine_k »

spike021 wrote: post #62
mnahkies wrote: post #56

That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point. A few things I've been thinking about recently: - we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier. - logging an error as a separate log line to the request…

If your codebase has the concept of a request ID, you could also feasibly use that to trace what a user has been doing with more specificity.

...if it does not, you should add it. A request ID, trace ID, correlation key, whatever you call it, you should thread it through every remote call, if you value your sanity.
Post reply on HN