Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tips for analyzing logs

jvns.ca

Post reply on HN
Print view View original

21–30 of 136 posts

Re: Tips for analyzing logs

#21

Post by francislavoie »

heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

At my current company, we're using pdsh with tail -f piped into sed to follow logs from multiple remote servers at the same time, label them and colorize them. Works okay. Decent solution without needing to install much other software. Not my favourite because it doesn't deal well with SSH sessions timing out and leaves some tail -f processes hanging, and some other quirks. But out of laziness and risk of breaking things in prod, we haven't tried experimenting with much else.

Re: Tips for analyzing logs

#23

Post by Severian »

My tips:

1) Fuck grep, use ripgrep, especially if you have to scour over an entire directory.

2) Get good with regex, seriously, it will shave hours off your searching.

3) For whatever application you are using, get to know how the logging is created. Find the methods used where said logs are made, and understand why such a log line exists.

4) Get good with piping into awk if needed if you need some nice readable output.

Re: Tips for analyzing logs

#24

Post by trashcan »

Also, sometimes logs stretch across multiple lines, and the other lines won't have the identifier you are searching for. For example, Java stack traces. In that case if you are stuck parsing unstructured logs, the simplest thing to do is to look at the entire file and search for the timestamp that found the first line.

Re: Tips for analyzing logs

#28

Post by burntsushi »

chasil wrote: post #16

A minor optimization is collapsing the grep -v, from this: cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4 to this: egrep -v 'THING1|THING2|THING3|THING4' file That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep. A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absor…

> A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absorbed into GNU grep with the -P option.

'pcregrep' still exists. But with PCRE2 supplanting PCRE, it is now spelled 'pcre2grep'.

I don't know the precise history of 'grep -P' and whether 'pcregrep' was actually absorbed into it, but 'pcregrep' is its own thing with its own features. For example, it has a -M/--multiline flag that no standard grep (that I'm aware of) has. (Although there are some work-arounds, e.g., by treating NUL as the line terminator via the -z/--null-data flag in GNU grep.)

Re: Tips for analyzing logs

#29

Post by UI_at_80x24 »

francislavoie wrote: post #21
heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

At my current company, we're using pdsh with tail -f piped into sed to follow logs from multiple remote servers at the same time, label them and colorize them. Works okay. Decent solution without needing to install much other software. Not my favourite because it doesn't deal well with SSH sessions timing out and leaves some tail -f processes hanging, and some other quirks. But out of laziness and risk of breaking th…

Run that in a tmux session. Problem solved.

Re: Tips for analyzing logs

#30

Post by phillipcarter »

Surprised to see that under the section "correlate between different systems" tracing isn't mentioned as an alternative approach. That's what tracing is: logging across different systems and getting that structure all stitched together for you.
Post reply on HN