Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tips for analyzing logs

jvns.ca

Post reply on HN
Print view View original

71–80 of 136 posts

Re: Tips for analyzing logs

#71

Post by aendruk »

amarshall wrote: post #67
digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

I do “useless” use of cat quite often because, in my brain, the pipeline naturally starts with “given this file”, so it makes the pipeline more consistent e.g. `cat f | a | b | c` rather than `a ` thread macro in Clojure, `|>` pipe in Elixir, and `&` reverse application operator in Haskell. If bash permitted putting the filename first, I’d stop using `cat`; alas, it does not.

There’s this. Though it would look nicer with another |.

Re: Tips for analyzing logs

#72

Post by amarshall »

aendruk wrote: post #71
amarshall wrote: post #67

Earlier quoted context omitted.

I do “useless” use of cat quite often because, in my brain, the pipeline naturally starts with “given this file”, so it makes the pipeline more consistent e.g. `cat f | a | b | c` rather than `a ` thread macro in Clojure, `|>` pipe in Elixir, and `&` reverse application operator in Haskell. If bash permitted putting the filename first, I’d stop using `cat`; alas, it does not.

There’s this. Though it would look nicer with another |.

That’s actually not half-bad. I think I’ll use that, thanks!

Re: Tips for analyzing logs

#73

Post by dllthomas »

digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

It's a lot more than the few extra cycles to spin up the process - it's also an extra copy of all the data. Usually that's also not much, but occasionally it's everything, as the consuming program can seek in a file, but not in a pipe, so might otherwise only need a tiny bit of the data.

Re: Tips for analyzing logs

#74

Post by marcosdumay »

digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

Well, you waste an entire fork and exec, so I believe you are underestimating the time by a few orders of magnitude. Also, it's almost always grep following the cat, so it's not much to memorize.

But it's well worth wasting a process to have a nice pipeline where each command does a single thing so you can easily reason about them.

Re: Tips for analyzing logs

#75

Post by anotherevan »

yakubin wrote: post #17
chasil wrote: post #16

A minor optimization is collapsing the grep -v, from this: cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4 to this: egrep -v 'THING1|THING2|THING3|THING4' file That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep. A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absor…

I usually prefer to pipe fgrep -v into fgrep -v. With egrep you need to escape brackets and other characters.

This also works:

fgrep -v -e THING1 -e THING2 -e THING3 -e THING4

Re: Tips for analyzing logs

#76

Post by anotherevan »

chasil wrote: post #16

A minor optimization is collapsing the grep -v, from this: cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4 to this: egrep -v 'THING1|THING2|THING3|THING4' file That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep. A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absor…

Or the following if pipes or egrep make you nervous:

grep -v -e THING1 -e THING2 -e THING3 -e THING4 file

Re: Tips for analyzing logs

#77

Post by chaps »

One thing I've done to identify infrequent log entries within a log file is to remove all numbers from a file and print out a frequency of each. Basically just helps to disregard timestamps (not just at the beginning of the line), line numbers, etc.

  cat file.log | sed 's/[0-9]//g' | sort | uniq -c | sort -nr
This has been incredibly helpful in quickly resolving outages more than once.

Re: Tips for analyzing logs

#78

Post by dj_gitmo »

heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

I love lnav and use it constantly, but it crashes a lot. I do wish there was something like lnav that was a little simpler to use, and written in a more resilient way that crashed less. I can cut lnav some slack for the crashes because identifying and parsing arbitrary log formats seems like a messy problem. Still it shouldn't crash 1/3rd of the time I use it.

Re: Tips for analyzing logs

#79

Post by baud147258 »

A few weeks ago I had a windows installer that was silently failing when upgrading from an older version (installation from scratch was working without issues). And as windows install logs aren't exactly easy to read, I was stumped, until I took an upgrade log from an older, working build, strip all date information from both files and compare them, checking all the sections which were different, until I found a line indicating that a colleague had forgotten about a limitation when dealing with msp (don't delete components on minor upgrades, but I didn't throw any stones as I've done the same mistake, twice, one and two years ago...)

Re: Tips for analyzing logs

#80

Post by nerdponx »

10000truths wrote: post #42
jldugger wrote: post #36

Honestly, the most amazing thing I did with logs was learn how to do subtraction. Any time you have multiple instances of a thing and only some of them are bad, you can easily find the problem (if anyone bothered to log it) by performing bad - good. The way you do this is by aggregating logs by fingerprints. Removing everything but punctuation is a generic approach to fingerprinting, but is not exactly human friendly…

Isn’t this basically what structured binary logs are? Instead of writing a string like `Error: foo timeout after ${elapsed_amt} ms` to the log, you write a 4-byte error code and a 4 byte integer for elapsed_amt. I know there are libraries like C++’s nanolog that do this for you, under the hood.

Maybe this is a silly question, but is there much value in a 4-byte binary code compared to a human-readable log with human-readable codes? Maybe size, but logfmt especially is not much less compact than binary data.
Post reply on HN