Tips for analyzing logs • Hacker News Board

Re: Tips for analyzing logs

#61

Post by infogulch » Tue, Dec 13, 2022, 9:39 PM UTC

digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

> the time [used to] explain the 'useless[ness]' .. of cat to someone .. is greater than the total time that their lifetime usage of cat was going to waste

If you look for situations like this they are surprisingly common.

> pet peeve

Re: Tips for analyzing logs

#62

Post by viraptor » Tue, Dec 13, 2022, 9:40 PM UTC

kamma4434 wrote: post #47

Jeez, this stuff is frontpage on HN? Sounds… pretty basic. I’m sure our AI overlords could produce deeper content.

People on HN have varied level of skills, and this is a well structured introduction to diving into logs. It already started conversation about better tooling. Let's celebrate today's lucky 10000 https://xkcd.com/1053/ rather than talk something down for being basic.

Re: Tips for analyzing logs

#63

Post by heinrichhartman » Tue, Dec 13, 2022, 9:40 PM UTC

heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Looks great, but I struggle to get this working with the two main log sources on my system:

1) systemd logs (binary data stored at /var/log/journal)

2) docker logs (from various containers running on the host)

Any hints on getting this working? I don't see this covered in the docs (https://docs.lnav.org/en/latest/formats.html).

Re: Tips for analyzing logs

#64

Post by emmelaich » Tue, Dec 13, 2022, 9:46 PM UTC

I found the histogram technique to be really helpful. Slight mod - I tend to sort reverse at the end of the pipeline (sort -rn); then |head is often more useful.

It's also good to have histograms by hour or day. I've hacked up scripts to do this but I should really make something better!

Re: Tips for analyzing logs

#65

Post by geocrasher » Tue, Dec 13, 2022, 9:55 PM UTC

One thing I didn't see was how to use GREP to view the lines before and after a match:

   grep regex /var/log/logfile -A5 #To view the next 5 lines
   grep regex /var/log/logfile -B5 #To view the previous 5 lines
   grep regex /var/log/logfile -05 #To view the 5 lines before *and* after the match

This is super handy to find out what happened just before a service crashed, for example.

Re: Tips for analyzing logs

#66

Post by phillipcarter » Tue, Dec 13, 2022, 9:58 PM UTC

kamma4434 wrote: post #47

Jeez, this stuff is frontpage on HN? Sounds… pretty basic. I’m sure our AI overlords could produce deeper content.

Yours is an unhelpful and non-constructive comment. Clearly a lot of people have been getting something out of the content in this post, as it's started several discussions.

Re: Tips for analyzing logs

#67

Post by amarshall » Tue, Dec 13, 2022, 10:04 PM UTC

digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

I do “useless” use of cat quite often because, in my brain, the pipeline naturally starts with “given this file”, so it makes the pipeline more consistent e.g. `cat f | a | b | c` rather than `a ` thread macro in Clojure, `|>` pipe in Elixir, and `&` reverse application operator in Haskell. If bash permitted putting the filename first, I’d stop using `cat`; alas, it does not.

Re: Tips for analyzing logs

#68

Post by rsdbdr203 » Tue, Dec 13, 2022, 10:06 PM UTC

tetha wrote: post #50

As much as I approve of a skillset to analyze local logs, but after a relatively small scale (10-20 systems), a central decent log aggregation like opensearch or ELK just brings so much value even on 1-3 nodes. It'd be one of the first changes I make to an infrastructure because it's so powerful. And its not just log searching and correlation value. At work, the entire discussion "oh but we need access to all servers…

Agreed! Centralizing logs is so helpful, but you don't know it until you've done it. Too many people rely upon grep, when a handy tool is just a download away. Plug for said too: https://log-store.com

Re: Tips for analyzing logs

#69

Post by civopsec » Tue, Dec 13, 2022, 10:06 PM UTC

digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

Doesn’t people like Rob Pike argue for `cat |` since it fits so orthogonally with the rest of the pipeline?

Re: Tips for analyzing logs

#70

Post by rsdbdr203 » Tue, Dec 13, 2022, 10:08 PM UTC

lowbloodsugar wrote: post #41

pantulis wrote: post #11

Earlier quoted context omitted.

I think there should be an counterpart to "log analysis" which is "logging strategies for your app". Which should be WHAT to log and WHEN. Stuff like, if you are exposing an HTTP endpoint, you should log the request URL and the time it took to serve it. Or if you are invoking an extenral service, you should log the response time of that service.

And you should produce a single line of output for each request that identifies all of the pertinent information. You can have more than one, for e.g. a thread dump, but there should be one that provides a complete summary. I've lived with apps that logged at each stage of the process as separate lines, and that's just not useful data when grepping for anomalies.

I hypothesize it's not useful because you're using grep. If you use a tool that can show you multiple lines all tied by a request ID, it becomes much more helpful.