31–40 of 136 posts
My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.
A minor optimization is collapsing the grep -v, from this: cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4 to this: egrep -v 'THING1|THING2|THING3|THING4' file That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep. A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absor…
> A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absorbed into GNU grep with the -P option. 'pcregrep' still exists. But with PCRE2 supplanting PCRE, it is now spelled 'pcre2grep'. I don't know the precise history of 'grep -P' and whether 'pcregrep' was actually absorbed into it, but 'pcregrep' is its own thing with its own features. For example, it has…
GNU grep is also linked to pcre, not pcre2.
# pcre2grep
bash: pcre2grep: command not found...
# yum install pcre2grep
Last metadata expiration check: 1:58:58 ago on Tue 13 Dec 2022 11:45:44 AM CST.
No match for argument: pcre2grep
Error: Unable to find a match: pcre2grep
# yum whatprovides pcre2grep
Last metadata expiration check: 2:09:25 ago on Tue 13 Dec 2022 11:45:44 AM CST.
Error: No matches found.
# rpm -qa | grep pcre2 | sort
pcre2-10.40-2.0.2.el9.x86_64
pcre2-syntax-10.40-2.0.2.el9.noarch
pcre2-utf32-10.40-2.0.2.el9.x86_64
# which grep
/usr/bin/grep
# ldd /usr/bin/grep | grep pcre
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007efc473c4000)A minor optimization is collapsing the grep -v, from this: cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4 to this: egrep -v 'THING1|THING2|THING3|THING4' file That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep. A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absor…
I usually prefer to pipe fgrep -v into fgrep -v. With egrep you need to escape brackets and other characters.
$ fgrep -h
fgrep: warning: fgrep is obsolescent; using grep -F
$ egrep -h
egrep: warning: egrep is obsolescent; using grep -E> Often log lines will include a request ID. Yes, always include a request id in every request structure you create and include it also in the response and print it. It would seem something obvious that everyone does by default but instead, no, it's not so obvious it seems.
Not so obvious. How to implement it without passing request id to all the functions, when they are unrelated to request/http ? Especially in languages without thread-locals such as javascript?
The way you do this is by aggregating logs by fingerprints. Removing everything but punctuation is a generic approach to fingerprinting, but is not exactly human friendly. For Java, log4j can use class in your logging pattern, and that plus log level is usually pretty specific.
Once you have a fingerprint, the rest is just counting and division. Over a specific time window, count the number of log events, for every finger print, for both good and bad systems. Then score every fingerprint as (1+ # of bad events) / (1 + # of good events) and everything at the top is most strongly bad. And the more often its logged, the further up it will be. No more lecturing people about "correct" interpretations of ERROR vs INFO vs DEBUG. No more "this ERROR is always logged, even during normal operations".
My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.
At my current company, we're using pdsh with tail -f piped into sed to follow logs from multiple remote servers at the same time, label them and colorize them. Works okay. Decent solution without needing to install much other software. Not my favourite because it doesn't deal well with SSH sessions timing out and leaves some tail -f processes hanging, and some other quirks. But out of laziness and risk of breaking th…
It works by uploading a stub that is an "actually portable executable" to the remote over ssh. lnav then talks to the stub to sync files back to the local host. I don't know the scale of the hosts or files, but if they're not too big, it might work fine.
Honestly, the most amazing thing I did with logs was learn how to do subtraction. Any time you have multiple instances of a thing and only some of them are bad, you can easily find the problem (if anyone bothered to log it) by performing bad - good. The way you do this is by aggregating logs by fingerprints. Removing everything but punctuation is a generic approach to fingerprinting, but is not exactly human friendly…
My tips: 1) Fuck grep, use ripgrep, especially if you have to scour over an entire directory. 2) Get good with regex, seriously, it will shave hours off your searching. 3) For whatever application you are using, get to know how the logging is created. Find the methods used where said logs are made, and understand why such a log line exists. 4) Get good with piping into awk if needed if you need some nice readable out…