Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tips for analyzing logs

jvns.ca

Post reply on HN
Print view View original

11–20 of 136 posts

Re: Tips for analyzing logs

#11

Post by pantulis »

kerblang wrote: post #5

> you’ll get overwhelmed by a million irrelevant messages because the log level is set to INFO I know this happens, but I think it's because programmers are abusing INFO. In principle it's reserved for messages that are informative at a level sys admins and a few others can make sense of and use. Unfortunately abuse often leads to "We turned INFO off" making it much harder to diagnose things after the fact.

I think there should be an counterpart to "log analysis" which is "logging strategies for your app". Which should be WHAT to log and WHEN.

Stuff like, if you are exposing an HTTP endpoint, you should log the request URL and the time it took to serve it. Or if you are invoking an extenral service, you should log the response time of that service.

Re: Tips for analyzing logs

#12

Post by gh123man »

janci wrote: post #8
Manjuuu wrote: post #7

> Often log lines will include a request ID. Yes, always include a request id in every request structure you create and include it also in the response and print it. It would seem something obvious that everyone does by default but instead, no, it's not so obvious it seems.

Not so obvious. How to implement it without passing request id to all the functions, when they are unrelated to request/http ? Especially in languages without thread-locals such as javascript?

Many frameworks solve this with logger context. Add the properties you want to the logging context and all future logs in that context will have that property.

One simple example - Serilog enrichment https://github.com/serilog/serilog/wiki/Enrichment

This does however assume your entire application uses the same logger. But this is generally a requirement in services that require tracing like this.

Re: Tips for analyzing logs

#13

Post by sisk »

Loosely related: a few years ago I wanted a simpler alternative to some of the more feature-full log viewers out there so I threw together a tiny (50kb) app that might be useful to some folks in here.

All it does is consistently colors the first field in a line from stdin so you can quickly see which log lines have the same first field.

I used it in combination with the parallel[0] command to prefix log lines by replica name when tailing logs across machines: https://github.com/jasisk/color-prefix-pipe

[0]: https://www.gnu.org/software/parallel/

Re: Tips for analyzing logs

#15

Post by terom »

Re timing, logs like nginx access logs have their timestamp from when the request completed, not when the request came in. That's a significant difference for long duration (~10s+) requests, and matters when trying to correlate logs or metrics to a request.

Re: Tips for analyzing logs

#16

Post by chasil »

A minor optimization is collapsing the grep -v, from this:

  cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4
to this:

  egrep -v 'THING1|THING2|THING3|THING4' file
That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep.

A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absorbed into GNU grep with the -P option.

Re: Tips for analyzing logs

#17

Post by yakubin »

chasil wrote: post #16

A minor optimization is collapsing the grep -v, from this: cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4 to this: egrep -v 'THING1|THING2|THING3|THING4' file That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep. A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absor…

I usually prefer to pipe fgrep -v into fgrep -v. With egrep you need to escape brackets and other characters.

Re: Tips for analyzing logs

#19

Post by oefrha »

heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Huh, I almost posted a duplicate recommendation. My only complaint with lnav was that it had to be built from source on Linux and the build was frigging huge. Apparently they have a pre-compiled linux-musl binary now.

Re: Tips for analyzing logs

#20

Post by chasil »

yakubin wrote: post #17
chasil wrote: post #16

A minor optimization is collapsing the grep -v, from this: cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4 to this: egrep -v 'THING1|THING2|THING3|THING4' file That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep. A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absor…

I usually prefer to pipe fgrep -v into fgrep -v. With egrep you need to escape brackets and other characters.

True, for absolute fixed strings, the author's approach is superior.

Off the cuff, another way to do it is with awk's index function. I don't know what speed penalty this might impose.

  $ cat ~/fmgrep
  #!/bin/awk -f

  BEGIN { split(ARGV[1], s, ","); ARGV[1]="" }

  { m = 1; for(c in s) if(index($0, s[c])) m = 0; if(m) print }

  $ ~/fmgrep sshd,systemd,polkitd /var/log/secure
Post reply on HN