21–30 of 81 posts
One question I always have about logging: how do I log valid and expected but prohibited actions? That is, the system is behaving as designed but the user is seeing an error message because they're using the system wrong, and I want to know how often this is happening?
I fight log infra all the time. I can't win the fight against structured logging anymore, so I'm now fighting against type systems and allergies to global state to make log output available everywhere. If you're going to ram structured logging down devs' throats, then the least you can do is to make it easy enough to use. I don't want to have to pass a logging object everywhere. There's like two pieces of information…
On the levelled logging point, I stopped using levels after switching from Java -> Go and haven't looked back: https://thomshutt.github.io/opinionated-logging-in-go.html
IMO, the entire point of logs is to be able to ask questions of and reason about the current state of your application. If you're only logging errors that you can't recover from you may as well just throw and exception and restart.
Earlier quoted context omitted.
Not as much as you'd think, and critically, the cost is largely disconnected from how you use the infra.
If your infra is not on-prem, yes it will cost you more money as you are generating more and more and bigger logs.
On top of that, mechanical hard drives are pretty cheap these days. Like it's a dozen dollars per terabyte, if not less.
I don't know, you're either producing just absurd amounts of logs, on the order of a hundred gigabytes a day plain text, at which sure, I guess you could probably log a bit less. Either that or you're operating at a scale with many millions of users where you should have income and be able to afford it.
... well, either that, or you're being fleeced.
Log frugality and log uniqueness are great concepts. +1 for mentioning both the financial and cognitive costs of excessive logging.
It seems the author is putting a heavy emphasis on trying to create readable logs. Finding the signal in the noise. I am biased, but I think this is a failure of the tools used to read the logs rather than the logs themselves. This is why I wrote LogViewPlus (https://www.logviewplus.com/).
Earlier quoted context omitted.
There are still going to be time and effort costs involved in scaling that infrastructure as your log volume increases
You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated. It's usually only when you involve ELK or something like that your logs start to get big. Which in turn is typically necessitated by over-complicated distributed software design. If you're at the scale where this actually matters and you're serving millions or…
This is a good point - a RAID array of a few HDDs/SSDs scales surprisingly far and is cheaper than many of the cloud services out there, though whether you can or can't use either approach probably relies on compliance requirements and such.
I will definitely add that logs can compress really well - to the point where it's been close to a year since I added Logrotate to a project that didn't have it before, for a pretty basic setup, and I haven't had the need to even look at how many archives are currently retained, given that the disk usage has changed very slightly. And that's for multiple systems that filled up the available storage in months previously.
Of course, my personal gripe is that most of the logging solutions out there are rather complex - something like Graylog feels like one of the simpler self-hostable options while still being fully featured, but in my experience anything that runs ES is really resource hungry. Sometimes it feels like MariaDB/PostgreSQL would be good enough for most of the simpler low logging volume setups out there - if you don't want to manage logs as files, want to ship them somewhere, but don't want the receiving system to be too complex either.
Unrelated to the content: I really like the phrasing of the title. Not "…you should follow", not the tired "best practices", simply "Things I do".
I also can’t stand: “You’re doing logging wrong” / “you’ve been doing logging wrong”
But then again as an industry we seem to like confident bullshitters