31–40 of 81 posts
I have found the extra structure and familiar semantics make it a lot easier to talk about what we log, how we log it and why.
Something I haven't seen discussed very widely: it feels like there's not only a balance needed in determine what to log vs what not to log, but also logging in a way that isn't a detriment to the readability of the code overall. Over time I've actually found myself logging less just because having to sorta mentally elide logging lines added to the cognitive overhead of reading & understanding code.
Unrelated to the content: I really like the phrasing of the title. Not "…you should follow", not the tired "best practices", simply "Things I do".
- What happened - When it happened? - Where it happened? - Why it happened? - What's the next step?
If your log doesn't answer at least the first 3 questions, then it's useless. If you don't answer "why", then you should think harder whether that is useful or not.
If I had a cent for every time I see "Something went wrong" optionally followed by stack trace that is nearly entirely in 3rd party code with zero information to correlate it with anything - I would have retired to homestead ages ago.
- most devs have lost the concept of logging levels, considering normal spitting out crappy giants backtraces and wall of meaningless text;
- most devs have lost the idea of "being quiet" or "frugality", also have NEVER tried to read logs like an application user who do not have nor want to pass gazillion of lines of sources often crappily arranged.
In the classic *nix world skimming log for "alerting patterns" was easy, for modern crapplications it's a bit of a nightmare. Similarly using logs for debug and mere health check is sometimes useless since many messages should at maximum be debug level logs, others are meaningless and even looking at them from the sources does NOT clarify until you read much more.
I fight log infra all the time. I can't win the fight against structured logging anymore, so I'm now fighting against type systems and allergies to global state to make log output available everywhere. If you're going to ram structured logging down devs' throats, then the least you can do is to make it easy enough to use. I don't want to have to pass a logging object everywhere. There's like two pieces of information…
What is the reason structured logging is bad? I'm curious, as I felt like it made my life a lot easier.
On the levelled logging point, I stopped using levels after switching from Java -> Go and haven't looked back: https://thomshutt.github.io/opinionated-logging-in-go.html
This completely overlooks many things such as logs used as metrics or even just using them to reason about the state of your application. What if your application isn't throwing errors but some thing is still broke Or you're shipping bad data? IMO, the entire point of logs is to be able to ask questions of and reason about the current state of your application. If you're only logging errors that you can't recover fro…
Unironically tho this is a good decision if you design for it from the beginning crash-only software style. Most of what I log is INFO and WARN level because unexpected combinations of business-level state are where the real subtle nasty bugs are. Nil reference or whatever can just crash who cares.
We use SQLite for logging all the things. This sidesteps entire rabbit colonies worth of issues - especially with regard to downstream parsing & reporting. I have found the extra structure and familiar semantics make it a lot easier to talk about what we log, how we log it and why.
* Are you working on a SAAS product or embedded/IoT project or hobby project?
* How do you aggregate the SQLite logs together from disparate machines? Seems like you probably can't use fluentbit/filebeat/etc.
* Where do you query these logs?
* How do you structure these logs? (timestamp, machine, message) or something with more columns?
* Are you able to capture stacktraces?
I _love_ the idea of leveraging SQLite for this kind of scenario and possibly skipping a lot of messy plumbing or pricy vendors, but I'm uncertain how this works.
I think all of this sounds fine in theory, but the reality is that most logged information will not be needed - ever. Exactly what information is needed when can be difficult to predict. So, if a developer feels something might be important, they should probably log it. Within reason, I think it is better to have it and not need it than to need it and not have it. It seems the author is putting a heavy emphasis on tr…
Log retention starts to mess with what's considered reasonable. For example, despite the fact that it'd be actually legitimately useful to store 180 days of pcaps, that's just cost prohibitive.
Earlier quoted context omitted.
If your infra is not on-prem, yes it will cost you more money as you are generating more and more and bigger logs.
You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data. On top of that, mechanical hard drives are pretty cheap these days. Like it's a dozen dollars per terabyte, if not less. I don't know, you're either producing just absurd amounts of logs, on the order of a hundred gigabytes a day plain text, at w…
Services like CloudWatch are an excellent way to burn through money, though it's usually the time series storage and ingestion costs that balloon out of control.