Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Logging sucks

loggingsucks.com

Post reply on HN
Print view View original

131–140 of 232 posts

Re: Logging sucks

#131

Post by golem14 »

cowsandmilk wrote: post #14

Horrid advice at the end about logging every error, exception, slow request, etc if you are sampling healthy requests. Taking slow requests as an example, a dependency gets slower and now your log volume suddenly goes up 100x. Can your service handle that? Are you causing a cascading outage due to increased log volumes? Recovery is easier if your service is doing the same or less work in a degraded state. Increasing…

For high volume services, you can still log a sample of healthy requests, e.g., trace_id mod 100 == 0. That keeps log growth under control. The higher the volume, the smaller percentage you can use.

Re: Logging sucks

#132

Post by etamponi »

> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue. Your logs are still acting like it's 2005.

Perhaps it's time to take back the good things from 2005.

Re: Logging sucks

#133

Post by 0xbadcafebee »

> Here's the mental model shift that changes everything: Instead of logging what your code is doing, log what happened to this request.

Yeah that doesn't magically fix everything. Logging is still an arbitrary, clunky, unintuitive process that requires intentional design and extra systems to be useful.

The "Wide Event log" example is 949 bytes, which isn't unmanageably large, but it is 3x larger than most log messages which are about 300 bytes. And in that blob of data might be key insights, but it is left up to an extra engineering process to discover what might be unusual in that blob. It lacks things like code line numbers, stack trace, and context given by the program about its particular functions (rather than assumptions based on a few pieces of metadata). And it's excessively verbose, as it has a trace and request ID and service name, but duplicates information already available to tracing systems based on those 3 metrics.

> Wide events are a philosophy: one comprehensive event per request, with all context attached.

That's simply impossible. You cannot have all context from viewing a single point in the network, regardless of how hard you try to record or pass on information. That's the whole point of tracing: you correlate the context of different network points, specifically because that's the only way to discover the missing details.

> Modern columnar databases (ClickHouse, BigQuery, etc.) are specifically designed for high-cardinality, high-dimensionality data. The tooling has caught up. Your practices should too.

You should not depend on a space shuttle to get to the grocery store. Logging is intended to be an abstracted component which can be built on by other systems. Your app should work just as well running from Docker on your laptop as it does in the cloud.

Re: Logging sucks

#134

Post by otterley »

dpark wrote: post #127
otterley wrote: post #96

Earlier quoted context omitted.

At the end of the day, it comes down to what sort of functionality you want out of your observability. Modest needs usually require modest resources: sure, you could just append to log files on your application hosts and ship them to a central aggregator where they're stored as-is. That's cheap and fast, but you won't get a lot of functionality out of it. If you want more, like real-time indexing, transformation, ana…

Surely you aren’t doing real time indexing, transformation, analytics, etc in the same service that is producing the logs. A catastrophic increase in logging could certainly take down your log processing pipeline but it should not create cascading failures that compromise your service.

Of course not. Worst case should be backpressure, which means processing, indexing, and storage delays. Your service might be fine but your visibility will be reduced.

Re: Logging sucks

#135

Post by ivan_gammel »

dexwiz wrote: post #81
ivan_gammel wrote: post #68

Earlier quoted context omitted.

…and the same ID can be displayed to user on HTTP 500 with the support contact, making life of everyone much easier.

I have seen pushback on this kind of behavior because "users don't like error codes" or other such nonsense. UX and Product like to pretend nothing will ever break, and when it does they want some funny little image, not useful output. A good compromise is to log whenever a user would see the error code, and treat those events with very high priority.

Nah, that’s easy problem to solve with UX copy. „Something went wrong. Try again or contact support. Your support request number is XXXX XXXX“ (base 58 version of UUID).

Re: Logging sucks

#136

Post by dxdm »

0x3f wrote: post #109
dxdm wrote: post #100

Earlier quoted context omitted.

> If a user request is hitting that many things, in my view, that is a deeply broken architecture. Things can add up quickly. I wouldn't be surprised if some requests touch a lot of bases. Here's an example: a user wants to start renting a bike from your public bike sharing service, using the app on their phone. This could be an app developed by the bike sharing company itself, or a 3rd party app that bundles mobilit…

> These do not have to be separate services, but they are separate enough to warrant it. All of this arises from your failure to question this basic assumption though, doesn't it?

> All of this arises from your failure to question this basic assumption though, doesn't it?

Haha, no. "All of this" is a scenario I consider quite realistic in terms of what needs to happen. The question is, how should you split this up, if at all?

Mind that these concerns will be involved in other ways with other requests, serving customers and internal users. There are enough different concerns at different levels of abstraction that you might need different domain experts to develop and maintain them, maybe using different programming languages, depending on who you can get. There will definitely be multiple teams. It may be beneficial to deploy and scale some functions independently; they have different load and availability requirements.

Of course you can slice things differently. Which assumptions have you questioned recently? I think you've been given some material. No need to be rude.

Re: Logging sucks

#137

Post by Aurornis »

the_mitsuhiko wrote: post #25
heinrichhartman wrote: post #4

A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept. Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports gen…

> A post on this topic feels incomplete without a shout-out to Charity Majors I concur. In fact, I strongly recommend anyone who has been working with observability tools or in the industry to read her blog, and the back story that lead to honeycomb. They were the first to recognize the value of this type of observability and have been a huge inspiration for many that came after.

> They were the first to recognize the value of this type of observability

With all due respect to her great writing, I think there’s a mix of revisionist history blended with PR claims going on in this thread. The blog has some good reading, but let’s not get ahead of ourselves in rewriting history around this one person/company.

Re: Logging sucks

#139

Post by groundzeros2015 »

> An era of monoliths, single servers, and problems you could reproduce locally.

Actually this is the problem. It’s extremely difficult to debug when you do not preserve basic properties that allow you to read code and reason about events.

Re: Logging sucks

#140

Post by KaiserPro »

I agree that logging suck bollocks, especially when most of the time you really want metrics.

Opentel is great, but sadly A lot of the stuff that I am using doesn't support it.

The thing that made it much more bearable, even easy is loki and a decent log parser.

I know a lot of kids like using SQL to interact with things, loki's explore interface beats the living shit out of SQL (In my opinion) its really simple to just isolate and slice logs interactivly. You can build your query really simply.

It beats splunk/sumo/scuba(facebook's log system) hands down in terms of searchability.

Post reply on HN