The substance of this post is outstanding. The framing is not, though. Why does it have to sound so dramatic and provocative? It’s insulting to its audience. Grumpiness, in the long term, is a career-limiting attitude.
I get the AI feeling from it.
Logging sucks
21–30 of 232 posts
Re: Logging sucks
#22Horrid advice at the end about logging every error, exception, slow request, etc if you are sampling healthy requests. Taking slow requests as an example, a dependency gets slower and now your log volume suddenly goes up 100x. Can your service handle that? Are you causing a cascading outage due to increased log volumes? Recovery is easier if your service is doing the same or less work in a degraded state. Increasing…
It’s an important architectural requirement for a production service to be able to scale out their log ingestion capabilities to meet demand.
Besides, a little local on-disk buffering goes a long way, and is cheap to boot. It’s an antipattern to flush logs directly over the network.
Re: Logging sucks
#23Some excellent points raised in this article.
Re: Logging sucks
#24The substance of this post is outstanding. The framing is not, though. Why does it have to sound so dramatic and provocative? It’s insulting to its audience. Grumpiness, in the long term, is a career-limiting attitude.
I get the AI feeling from it.
It might have been AI-assisted, and it might not have been. It doesn’t really matter. The author is ultimately responsible for the end result.
Re: Logging sucks
#25A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept. Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports gen…
> A post on this topic feels incomplete without a shout-out to Charity Majors
I concur. In fact, I strongly recommend anyone who has been working with observability tools or in the industry to read her blog, and the back story that lead to honeycomb. They were the first to recognize the value of this type of observability and have been a huge inspiration for many that came after.
Re: Logging sucks
#26This article is attacking a strawman. It makes up terrible logs and then says they are bad. Even if this was a single monolith the logs still don't include even something like a thread id, to avoid mixing different requests together.
Re: Logging sucks
#27Horrid advice at the end about logging every error, exception, slow request, etc if you are sampling healthy requests. Taking slow requests as an example, a dependency gets slower and now your log volume suddenly goes up 100x. Can your service handle that? Are you causing a cascading outage due to increased log volumes? Recovery is easier if your service is doing the same or less work in a degraded state. Increasing…
Yea that was my thought too. I like the idea in principle, but these magic thresholds can really bite you. It claims to be P(99), probably off some historical measurement, but that's only true if it's dynamically changing. Maybe this could periodically query the OTEL provider for the real number to at least limit the time window of something bad happening.
Re: Logging sucks
#28I've generally found that structured logs that include a correlation ID make it quite easy to narrow down the general area or exact cause of problems. Usually (in enterprise orgs) via Splunk or Datadog.
Where I've had problems it's usually been one of:
There wasn't anything logged in the error block. A comment saying "never happens" is often discovered later :)
Too much was logged and someone mandated dialing the logging down to save costs. Sigh.
A new thread was started and the thread-local details including the correlation ID got lost, then the error occurred downstream of that. I'd like better solutions for that one.
Edit: Incidentally a correlation ID is not (necessarily) the same thing as a request ID. An API often needs to allow for the caller making multiple calls to achieve an objective; 5 request IDs might be tied to a single correlation ID.
Re: Logging sucks
#29Slapping on OpenTelemetry actually will solve your problem.
Point #1 isn't true, auto instrumentation exists and is really good. When I integrate OTel I add my own auto instrumentors wherever possible to automatically add lots of context. Which gets into point #2.
Point #2 also isn't true. It can add business context in a hierarchal manner and ship wide events. You shouldn't have to tell every span all the information again. Just where it appears naturally the first time.
Point #3 also also isn't true because OTel libs make it really annoying to just write a log message and very strongly pushes you into a hierarchy of nested context managers.
Like the author's ideal setup is basically using OTel with Honeycomb. You get the querying and everything. And unlike rawdogging wide events all your traces are connected, can span multiple services and do timing for you.
Re: Logging sucks
#30[deleted]