the best implementation of structured logging I've seen is dotnet build's binlogs (https://msbuildlog.com), I would love to see it evolve into a general purpose logging solution
Logging sucks
101–110 of 232 posts
Re: Logging sucks
#102distributed event id and you are all set
Re: Logging sucks
#103"Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally."
But the next era will be like the previous one. Today monolith is enough for most of apps.
Re: Logging sucks
#104Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue.
If a user request is hitting that many things, in my view, that is a deeply broken architecture.
I'm building an analytics SaaS and we made the conscious decision to keep it simple: Next.js API routes + Supabase + minimal external services. A single page view hits maybe 3 components max (CDN -> App -> Database).
That said, I agree completely on structured logging with rich context. We include user_id, session_id, and event_type on every log line. Makes debugging infinitely easier.
The "wide events" concept is solid, but the real win is just having consistent, searchable structure. You don't need a revolutionary new paradigm - just stop logging random strings and use JSON with a schema.
Re: Logging sucks
#105Persisting a data schema that represents business events is a great idea. That’s more about Event Sourcing though and doing that can answer a ton of questions about the system without doing it in log messages.
Wide events as a strategy is expensive, even with sampling, and doesn’t address the fundamental problem - why do we log messages?
I was hoping the article would enumerate why we log messages. Nailing down those scenarios first will lead to a happy life.
Why do we log? - proof of life - is the system running? - what is the state (in memory) when an error occurred? - when did an error occur? - do I need to get up at 2 am and fix something? - what do I need to fix?
I feel like every team operating a system has their own reasons for logging.
Re: Logging sucks
#106This seems like a classic time vs space trade off.
Instead of reconstructing a "wide event" from multiple log lines with the same request id, the suggestion seems to be logging wide events repeatedly to simplify reconstruction from request ids.
I personally don't see the advantage, and in either scenario, if you're not logging what's needed your screwed.
Re: Logging sucks
#107Earlier quoted context omitted.
> but it's a reality we're facing. Yes. Most software is bad The incentives between managers and technicians are all wrong Bad software is more profitable, over the time frames managers care about, than good software
The reason we end up with very complex systems I don't think is because of incentives between "managers and technicians". If I were to put my finger to it, I would assume it's the very technicians who argued themselves into a world where increased complexity and more dependencies is seen as a good thing. Fighting complexity is deeply unpopular.
At least in my place of work, my non-technical manager is actually on board with my crusade against complex nonsense. Mostly because he agrees it would increase feature velocity to not have to touch 5 services per minor feature. The other engineers love the horrific mess they've built. It's almost like they're roleplaying working at Google and I'm ruining the fun.
Re: Logging sucks
#108Horrid advice at the end about logging every error, exception, slow request, etc if you are sampling healthy requests. Taking slow requests as an example, a dependency gets slower and now your log volume suddenly goes up 100x. Can your service handle that? Are you causing a cascading outage due to increased log volumes? Recovery is easier if your service is doing the same or less work in a degraded state. Increasing…
Just implement exponential backoff for slow requests logging, or some other heuristic, to control it. I definitely agree it is a concern though.
Re: Logging sucks
#109> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue. Your logs are still acting like it's 2005. If a user request is hitting that many things, in my view, that is a deeply broken architecture.
> If a user request is hitting that many things, in my view, that is a deeply broken architecture. Things can add up quickly. I wouldn't be surprised if some requests touch a lot of bases. Here's an example: a user wants to start renting a bike from your public bike sharing service, using the app on their phone. This could be an app developed by the bike sharing company itself, or a 3rd party app that bundles mobilit…
> These do not have to be separate services, but they are separate enough to warrant it.
All of this arises from your failure to question this basic assumption though, doesn't it?
Re: Logging sucks
#110> No grep-ing.
How is grep a bad thing? I find myself using it all the time.
I’m not into graphical user interfaces. They overwhelm me. By the time I’ve clicked myself through the GUI or written some horrible proprietary $COMPANY Query Language string, I might have already figured out the bug using tried and tested CLI tools.