Use events instead of repetitious logging calls.
https://dave.autonoma.ca/blog/2022/01/08/logging-code-smell/
Logging sucks
111–120 of 232 posts
Re: Logging sucks
#112> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. I worked with enterprise message bus loggers in semiconductor manufacturing context wherein we had thousands of participants on the message bus. It generated something like 300-400 megabytes per hour. Despite the insane volume we made this work really well using just grep and other basic CLI tools.…
> It generated something like 300-400 megabytes per hour. Despite the insane volume we made this work really well using just grep and other basic CLI tools.
400MB of logs an hour is nothing at all, that's why a naive grep can work. You don't even need to rotate your log files frequently in this situation.
Re: Logging sucks
#113I've recently added error tracking to my self-hosted analytics app (UXWizz), and the way I did it is simply add extra events to each user/session. Once you have the concept of a session or user, you can simply attach errors or logs as Events stored for that user. This solves the main problem mentioned in the article, where you don't know what happened, plus being an Event stored in a MySQL database, you can still query it.
Why not simply use Events for logging, instead of plain strings?
Re: Logging sucks
#114A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept. Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports gen…
This post was so in-line with her writing that I was really expecting it to turn into an ad for Honeycomb at the end. I was pretty surprised with it turned out the author was unaffiliated!
Re: Logging sucks
#115The substance of this post is outstanding. The framing is not, though. Why does it have to sound so dramatic and provocative? It’s insulting to its audience. Grumpiness, in the long term, is a career-limiting attitude.
Career-limiting perhaps (if expressing normal human emotion is a minus inside of an organization, it may be time to bail) but some of the best minds I've met/observed were absolute curmudgeons (with purpose—they were properly bothered by a problem and refused to go along with the "sweep it under the rug" behavior).
Sure, I've dealt with plenty of assholes, too, but the grumps are usually just tired of their valid insight being ignored by more foolish, orthogonally incentivized types (read: "playing the game" not "making it work well").
Re: Logging sucks
#116> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. I worked with enterprise message bus loggers in semiconductor manufacturing context wherein we had thousands of participants on the message bus. It generated something like 300-400 megabytes per hour. Despite the insane volume we made this work really well using just grep and other basic CLI tools.…
At last a sane person. Logs are for identifying the event timeline, not to acquire the whole reqs/resp data. Putting every detail into the logs is -in my experience - makes undertanding issues harder. Logs tell a story. When, what happened, not how or why that happened. Why is in the code, how is in the combination of, data, logs, events, code.
And loosely related, I also dislike log interfaces like elk stack. They make following track of events really hard. Most of the time you do not know what you are loooking for, just a vauge understanding of why you are looking into the logs. So a line passed 3 micro seconds ago maybe your euraka moment, where no search could identify , just intuition and following logs diligently can.
Re: Logging sucks
#117While I agree with some of it, I feel like there's a big gotcha here that isn't addressed.
Having 1 single wide event, at the end of a request, means that if something unexpected happens in the middle (stack overflow, some bug that throws an error that bypasses your logging system, lambda times out etc...) you don't get any visibility into what happens.
You also most likely lose out on a lot of logging frameworks your language has that your dependencies might use.
I would say this is a good layer to put on top of your regular logs. Make sure you have a request/session wide id and aggregate all those in your clickhouse or whatever into a single "log".
Re: Logging sucks
#118I've recently come off a team that was racking up a huge Splunk bill with ~70 log events for each request on a high traffic service, and this is all very resonant (except the bit about sampling, I never gave that much thought - reducing our Splunk bill 70x was ambitious enough for me!).
Hadn't heard the "wide event" name, but I had settled on the same idea myself in that time (called them "top-level events" - i.e. we would gather information from the duration of the request and only log it at the "top" of the stack at the end), and evangelised them internally mostly on the basis it gave you fantastic correlation ability.
In theory if you've got a trace id in Splunk you can do correlated queries anyway, but we were working in Spring and forever having issues with losing our MDC after doing cross-thread dispatch and forgetting to copy the MDC thread global across. This wasn't obvious from the top-level, and usually only during an incident would you realise you weren't seeing all the loglines you expected for a given trace. So absent a better solution there, tracking debug info more explicitly was appealing.
Also used these top-level events to store sub-durations (e.g. for calling downstream services, invoking a model etc), and with Splunk if you record not just the length of a sub-process but its absolute start, you can reconstruct a hacky waterfall chart of where time was spent in your query.
Re: Logging sucks
#119Horrid advice at the end about logging every error, exception, slow request, etc if you are sampling healthy requests. Taking slow requests as an example, a dependency gets slower and now your log volume suddenly goes up 100x. Can your service handle that? Are you causing a cascading outage due to increased log volumes? Recovery is easier if your service is doing the same or less work in a degraded state. Increasing…
Good point. It also reminded me of when I was trying to optimize my app for some scenarios, then I realized it's better to optimize it for ALL scenarios, so it works fast and the servers can handle no matter what. To be more specific, I decided NOT to cache any common queries, but instead make sure that all queries are fast as possible.
Re: Logging sucks
#120The substance of this post is outstanding. The framing is not, though. Why does it have to sound so dramatic and provocative? It’s insulting to its audience. Grumpiness, in the long term, is a career-limiting attitude.
Career-limiting perhaps (if expressing normal human emotion is a minus inside of an organization, it may be time to bail) but some of the best minds I've met/observed were absolute curmudgeons (with purpose—they were properly bothered by a problem and refused to go along with the "sweep it under the rug" behavior). Sure, I've dealt with plenty of assholes, too, but the grumps are usually just tired of their valid ins…
We've all tolerated the grumpy genius at some point in our careers. Nevertheless, most of us would prefer to work with a person who's both smart and kind over someone who's smart and curmudgeonly. It is possible to be both smart and kind, and I've had the pleasure of working with such people.
Assholes can sap an organization's strength faster than any productive value their intelligence can provide. I'm not suggesting the author is an asshole, though; there's not enough evidence from this post.