Tips for analyzing logs

#1

Post by historynops » Tue, Dec 13, 2022, 4:47 PM UTC

Re: Tips for analyzing logs

#2

Post by heliostatic » Tue, Dec 13, 2022, 5:11 PM UTC

My biggest quality of life improvement for understanding logs has been lnav (https://lnav.org/) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Re: Tips for analyzing logs

#3

Post by [deleted] » Tue, Dec 13, 2022, 5:19 PM UTC

[deleted]

Re: Tips for analyzing logs

#4

Post by clumsysmurf » Tue, Dec 13, 2022, 5:20 PM UTC

heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Interesting! Do you know how it compares to multitail? (https://www.vanheusden.com/multitail/) They look very similar.

Re: Tips for analyzing logs

#5

Post by kerblang » Tue, Dec 13, 2022, 5:20 PM UTC

> you’ll get overwhelmed by a million irrelevant messages because the log level is set to INFO

I know this happens, but I think it's because programmers are abusing INFO. In principle it's reserved for messages that are informative at a level sys admins and a few others can make sense of and use. Unfortunately abuse often leads to "We turned INFO off" making it much harder to diagnose things after the fact.

Re: Tips for analyzing logs

#6

Post by kqr » Tue, Dec 13, 2022, 5:30 PM UTC

One thing that's improved my log analysis is learning awk (well actually Perl but I think 95 % of what I do with Perl I could also do with just awk). Often the most useful way to look at logs is statistically, and awk can quickly let you aggregate things like time-between-statements, counts, rates, state transition probabilities, etc for arbitrary patterns.

Re: Tips for analyzing logs

#7

Post by Manjuuu » Tue, Dec 13, 2022, 5:47 PM UTC

> Often log lines will include a request ID.

Yes, always include a request id in every request structure you create and include it also in the response and print it. It would seem something obvious that everyone does by default but instead, no, it's not so obvious it seems.

Re: Tips for analyzing logs

#8

Post by janci » Tue, Dec 13, 2022, 6:00 PM UTC

Manjuuu wrote: post #7

> Often log lines will include a request ID. Yes, always include a request id in every request structure you create and include it also in the response and print it. It would seem something obvious that everyone does by default but instead, no, it's not so obvious it seems.

Not so obvious. How to implement it without passing request id to all the functions, when they are unrelated to request/http ? Especially in languages without thread-locals such as javascript?

Re: Tips for analyzing logs

#9

Post by tstack » Tue, Dec 13, 2022, 6:00 PM UTC

clumsysmurf wrote: post #4

heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Interesting! Do you know how it compares to multitail? ( https://www.vanheusden.com/multitail/ ) They look very similar.

I don't think multitail really understands logs like lnav does, it's just following the last lines in the file. For example, if you try to follow multiple files in multitail like so:

    $ multitail /var/log/install.log -I /var/log/system.log

You get a view with the tail from one file followed by the tail from the other, they are not collated by timestamp. In contrast, if you do the same thing in lnav:

    $ lnav /var/log/install.log /var/log/system.log

You will get a single view with all of the log messages from both files and they're sorted by their timestamps. Here is all of what lnav is doing:

  * Monitoring files/directories for updates
  * Decompressed files/archives
  * Detected the log format for each file
  * Created SQLite vtables that provide access to log messages
  * Built an index of all the log messages in all the files so
    you can jump to a certain point in time or to the
    next/previous error message.
  * Display all log messages with syntax highlighting

Re: Tips for analyzing logs

#10

Post by TeMPOraL » Tue, Dec 13, 2022, 6:05 PM UTC

heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Thanks for linking to this!

Until now I thought logview.el[0] is the bee's knees, but now I can feel feature envy set in. There are some seriously powerful ideas listed on the lnav page, and it's also the first time I saw SQLite virtual tables used in the wild.

--

[0] - https://github.com/doublep/logview