No mention of klogg? It’s a lifesaver when opening huge log files
Tips for analyzing logs
111–120 of 136 posts
Re: Tips for analyzing logs
#112Your friends - grep (or ripgrep), sort, uniq, head, tail, cut, colrm, tr - diff - jq - bat - ministat - less, more, percol - Facebook Path Picker (fpp)
One of my favorites that I don't see used enough is:
watch -d -n $INT
I have lots of bad habits on the cli I should correct, now I'm feeling guilty after reading all the comments.
Re: Tips for analyzing logs
#113One of my favorite tricks is to use a visual difftool.
Copy good log into left panel. Copy bad log into right panel. Quickly show which lines are new, which are missing and which are out of order. Obviously ignore the timestamps ;)
Re: Tips for analyzing logs
#114I saw a tip a whilr back about not needing to keep adding "| grep -v stuff" but instead "grep -e -v stuff -v stuff2" i remember getting it to work on Linux but last i tried that on macos I didn't have much luck
Re: Tips for analyzing logs
#115One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…
It totally makes sense, after all the electron apps/docker containers/statically linked utilities/microservices running on my machine, needlessly running cat might be the straw that breaks the camels back.
Re: Tips for analyzing logs
#116Earlier quoted context omitted.
Isn’t this basically what structured binary logs are? Instead of writing a string like `Error: foo timeout after ${elapsed_amt} ms` to the log, you write a 4-byte error code and a 4 byte integer for elapsed_amt. I know there are libraries like C++’s nanolog that do this for you, under the hood.
Maybe this is a silly question, but is there much value in a 4-byte binary code compared to a human-readable log with human-readable codes? Maybe size, but logfmt especially is not much less compact than binary data.
Well, the value is mostly in query writing, but at that point you're basically writing a very bad TSDB?
Re: Tips for analyzing logs
#117> cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4
Keyboard manufacturers HATE this simple trick:
grep -vE "THING1|THING2|THING3|THING4" file
Re: Tips for analyzing logs
#118My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.
Looks great, but I struggle to get this working with the two main log sources on my system: 1) systemd logs (binary data stored at /var/log/journal) 2) docker logs (from various containers running on the host) Any hints on getting this working? I don't see this covered in the docs ( https://docs.lnav.org/en/latest/formats.html ).
for systemd logs this works for me
journalctl -o json | lnavRe: Tips for analyzing logs
#119Earlier quoted context omitted.
I hypothesize it's not useful because you're using grep. If you use a tool that can show you multiple lines all tied by a request ID, it becomes much more helpful.
I am quite often looking for patterns across thousands of requests. As an example, one thing I inherited didn’t even log how long each request took to serve . Sure, you find out how long a single request took, by comparing the first and last log entry, but that’s just not useful 99% of the time.
This is exactly the kind of lack of logging info that exasperates any on call person.
Re: Tips for analyzing logs
#120One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…
It totally makes sense, after all the electron apps/docker containers/statically linked utilities/microservices running on my machine, needlessly running cat might be the straw that breaks the camels back.
not counting the subpixel aliasing for your terminal font ;)