Lots of logs can be replaced by metrics. People go crazy with logs.
Logging practices I follow
41–50 of 81 posts
Re: Logging practices I follow
#42Earlier quoted context omitted.
There are still going to be time and effort costs involved in scaling that infrastructure as your log volume increases
You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated. It's usually only when you involve ELK or something like that your logs start to get big. Which in turn is typically necessitated by over-complicated distributed software design. If you're at the scale where this actually matters and you're serving millions or…
And what are you going to do when you need a human to read sixteen trillion bytes of compressed logs streamed off a single SATA disk?
Once you face the fact that the performance of a single SATA disk means you can't search the logs in any quick time, and nobody can possibly read that much log data, so nobody will use it, you start to see it as a hoarding disorder not a useful tool.
Re: Logging practices I follow
#43On the levelled logging point, I stopped using levels after switching from Java -> Go and haven't looked back: https://thomshutt.github.io/opinionated-logging-in-go.html
Good point! Log levels are pretty useless most of the time.
I would add that there can be value in having 2 log levels: verbose and non-verbose. It is helpful if you can selectively switch on verbose logging by user or by API endpoint.
In one application which I maintain, when verbose logging is switched on for a particular user, TCP/UDP socket objects are automatically wrapped and packet captures are logged, only for packets sent/received while servicing that particular user's requests. This has been a lifesaver when debugging things like weird, transient authentication problems stemming from upstream providers.
Re: Logging practices I follow
#44Earlier quoted context omitted.
You have to output a lot of logs before you fill up even a single large consumer-grade hard drive, especially given logs are typically compressed when rotated. It's usually only when you involve ELK or something like that your logs start to get big. Which in turn is typically necessitated by over-complicated distributed software design. If you're at the scale where this actually matters and you're serving millions or…
And what are you going to do when you need a human to read sixteen trillion bytes of compressed logs streamed off a single SATA disk? Once you face the fact that the performance of a single SATA disk means you can't search the logs in any quick time, and nobody can possibly read that much log data, so nobody will use it, you start to see it as a hoarding disorder not a useful tool.
It's not unheard of to need to retain years even decades worth of logs due to regulatory compliance. Nobody is reading them, they just need to exist. In that scenario you'll probably keep the current year or so fresh on a mechanical drive and past years on tape.
Re: Logging practices I follow
#45One additional thing I like in structured logs is having some form of context level information be included eith your logs, so that you already know things like tenant id, user id, request id, and basic parameters of the request without having to rewrite all that everytime you get an exception. Unrelated: I live in the Pacific north west and I clicked on this expecting to find a list like "don't log old growth for ti…
Is there a structured form of tracing? Because I feel like this contextual information should easily be part of a trace.
Re: Logging practices I follow
#46One thing that's an absolute must: Put an ISO-8601 timestamp at the very beginning of every line in your log. No apache format, no "sun" or "jan" or other words. ISO only. Seriously.
If the timestamp is in a weird format (or, god help you, multiple formats since some libraries log shit in a special way), it'll be just about impossible to tell when things actually happened instead of just when the logging server saw them. In a perfect world these would be milliseconds apart, but lots of bad stuff can happen.
Your log-grepping-guy will thank you.
Re: Logging practices I follow
#47Lots of logs can be replaced by metrics. People go crazy with logs.
Very much (e.g. timing data, counts like success and error counts and operations should be metrics). If not metrics, then traces. If not metrics or traces, then business events (e.g. like alerts, audit records). Almost everything that people put in logs actually belongs somewhere else, in my opinion. Which is evidenced by so much of log processing being about turning logs back into whatever it was they were supposed to be in the first place (metrics, distributed traces and business events).
Re: Logging practices I follow
#48One additional thing I like in structured logs is having some form of context level information be included eith your logs, so that you already know things like tenant id, user id, request id, and basic parameters of the request without having to rewrite all that everytime you get an exception. Unrelated: I live in the Pacific north west and I clicked on this expecting to find a list like "don't log old growth for ti…
Is there a structured form of tracing? Because I feel like this contextual information should easily be part of a trace.
Logging is kinda a mess in general for contextualization. Most that support it use kV tupled appended to the log line itself. OpenTelemetry is probably the best hope of supporting a world with contextual logs, metrics, traces which imho is a good thing. OTEL logging does some opinionated things with message construction though, so caveat emptor.
Re: Logging practices I follow
#49Earlier quoted context omitted.
You actually have to log a damn lot to actually fill up even a single 16 Tb drive with gzip-compressed logs which typically have something like 50x compression for log data. On top of that, mechanical hard drives are pretty cheap these days. Like it's a dozen dollars per terabyte, if not less. I don't know, you're either producing just absurd amounts of logs, on the order of a hundred gigabytes a day plain text, at w…
When people complain about the cost of excessive logging, they are almost certainly not thinking in terms of how much a drive costs. Services like CloudWatch are an excellent way to burn through money, though it's usually the time series storage and ingestion costs that balloon out of control.
Well, also the kind of people who worry about this are not thinking in terms of "a terabyte", like GP. It's always easy to give advice when your experience has been at a toy level.
Re: Logging practices I follow
#50Earlier quoted context omitted.
What is the reason structured logging is bad? I'm curious, as I felt like it made my life a lot easier.
I have the same question! I understand parent's criticisms of context objections and logging boilerplate, but I'm not following the "fight against structured logging." What are the alternatives? No logging? Unstructured logging? Why would either of those be better than structured logs?
While I personally favor no logging, almost everyone who criticizes structured logging would prefer unstructured logging so they can make it someone else's job to restructure it (i.e. index and query it).