Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tips for analyzing logs

jvns.ca

Post reply on HN
Print view View original

121–130 of 136 posts

Re: Tips for analyzing logs

#121

Post by bornfreddy »

digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

> One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since.

Wear it as a badge of honor! It marks you as a person who puts clarity, convenience and simplicity before raw performance. I can't think of a single case when that bit of performance matters.

Needless to say, I'm happily using cat (uselessly) myself and have no plans to convert.

Re: Tips for analyzing logs

#122

Post by sofixa »

tetha wrote: post #50

As much as I approve of a skillset to analyze local logs, but after a relatively small scale (10-20 systems), a central decent log aggregation like opensearch or ELK just brings so much value even on 1-3 nodes. It'd be one of the first changes I make to an infrastructure because it's so powerful. And its not just log searching and correlation value. At work, the entire discussion "oh but we need access to all servers…

Just to expand on this more - you don't have to use ELK which is pretty resource and maintenance heavy, there are much easier and faster alternatives that don't need attentive care.

Throw Loki+Grafana on a single VM somewhere (or run it on your Kubernetes/Nomad/ECS/etc. cluster) and it will get you very far, as long as you plan ahead a bit (most notably, indexing happens at ingestion, so you need to have an idea of what you want from your logs or your queries will be slower).

Or use a SaaS like logz.io, AWS OpenSearch, Datadog, etc. Most support OpenTelemetry now, so switching data ingestion is quite easy (unlike dashboards and alerts).

It makes sense to have centralised logs pretty much as soon as you outgrow the "everything runs on this one box and my DR plan is a prayer" stage, IMO.

Re: Tips for analyzing logs

#123

Post by burntsushi »

chasil wrote: post #32
burntsushi wrote: post #28

Earlier quoted context omitted.

> A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absorbed into GNU grep with the -P option. 'pcregrep' still exists. But with PCRE2 supplanting PCRE, it is now spelled 'pcre2grep'. I don't know the precise history of 'grep -P' and whether 'pcregrep' was actually absorbed into it, but 'pcregrep' is its own thing with its own features. For example, it has…

Oddly, there are pcre2 packages in RedHat/Alma 9, but they do not include a pcre2grep. GNU grep is also linked to pcre, not pcre2. # pcre2grep bash: pcre2grep: command not found... # yum install pcre2grep Last metadata expiration check: 1:58:58 ago on Tue 13 Dec 2022 11:45:44 AM CST. No match for argument: pcre2grep Error: Unable to find a match: pcre2grep # yum whatprovides pcre2grep Last metadata expiration check:…

GNU grep recently migrated to PCRE2. My GNU grep is linked to PCRE2:

    $ grep --version | head -n2
    grep (GNU grep) 3.8
    Copyright (C) 2022 Free Software Foundation, Inc.
    $ ldd /usr/bin/grep
            linux-vdso.so.1 (0x00007ffd2ddd5000)
            libpcre2-8.so.0 => /usr/lib/libpcre2-8.so.0 (0x00007f4f88b81000)
            libc.so.6 => /usr/lib/libc.so.6 (0x00007f4f8899a000)
            /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f4f88c6f000)
As for pcre2grep, Archlinux includes it as part of the pcre2 package:

    $ pacman -Qo $(which pcre2grep)
    /usr/bin/pcre2grep is owned by pcre2 10.40-3
So this is a distro packaging thing. But what I said is true: pcregrep, pcre2grep and grep -P are all distinct things.

Re: Tips for analyzing logs

#124

Post by btschaegg »

bornfreddy wrote: post #121
digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

> One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Wear it as a badge of honor! It marks you as a person who puts clarity, convenience and simplicity before raw performance. I can't think of a single case when that bit of performance matters. Needless to say, I'm happily using cat (uselessly) myself and have no plans to conve…

> It marks you as a person who puts clarity, convenience and simplicity before raw performance.

This. As noted, even in scripts it usually makes more sense since the result is a pipeline that's easier to read, annotate and modify.

Case in point:

  cat file.txt \
  | sed '1s/^\xEF\xBB\xBF//' `# Strip UTF-8 BOM at the beginning of the file` \
  | ...
Specifying a file name would only make the "black-magic-line" of `sed` more complicated while also making it more complicated to modify the pipeline itself. Now, if I want to skip that step to test something, I don't have to figure out if/how the next command takes an input file (or, ironically, replace `sed` with `cat`).

Re: Tips for analyzing logs

#125

Post by jpitz »

hayst4ck wrote: post #97
jpitz wrote: post #89

Earlier quoted context omitted.

I use ack for this purpose, because the color filtering options are so good. https://beyondgrep.com/

Every time I've done research (twice) ripgrep comes out on top. Usually it's something like rg > ag > ack > grep and that's overall testimonial, not just speed. What am I missing?

I started using ack about 14 years ago, so theres a lot of inertia there. It is slower than the others, but it was so much faster at the time for searching through source trees than grep.

For log colorizing, I make use of the --color-match and --passthru arguments, like https://powdahound.com/2009/10/colorize-log-output-with-ack/

Re: Tips for analyzing logs

#126

Post by burntsushi »

jpitz wrote: post #125
hayst4ck wrote: post #97

Earlier quoted context omitted.

Every time I've done research (twice) ripgrep comes out on top. Usually it's something like rg > ag > ack > grep and that's overall testimonial, not just speed. What am I missing?

I started using ack about 14 years ago, so theres a lot of inertia there. It is slower than the others, but it was so much faster at the time for searching through source trees than grep. For log colorizing, I make use of the --color-match and --passthru arguments, like https://powdahound.com/2009/10/colorize-log-output-with-ack/

For log colorizing, here's the approximate equivalent of that blog post for ripgrep:

    cat UNLICENSE 
      | rg --passthru --color always --colors 'match:fg:red' public 
      | rg --passthru --color always --colors 'match:fg:green' heirs 
      | rg --passthru --color always --colors 'match:fg:cyan' benefit

Re: Tips for analyzing logs

#127

Post by oftenwrong »

geocrasher wrote: post #65

One thing I didn't see was how to use GREP to view the lines before and after a match: grep regex /var/log/logfile -A5 #To view the next 5 lines grep regex /var/log/logfile -B5 #To view the previous 5 lines grep regex /var/log/logfile -05 #To view the 5 lines before *and* after the match This is super handy to find out what happened just before a service crashed, for example.

    grep regex /var/log/logfile -05 #To view the 5 lines before *and* after the match
This appears to work with GNU grep, at least, but YSK that the POSIX grep option for this is `-C`.

Re: Tips for analyzing logs

#129

Post by dj_gitmo »

tstack wrote: post #87
dj_gitmo wrote: post #78

Earlier quoted context omitted.

I love lnav and use it constantly, but it crashes a lot. I do wish there was something like lnav that was a little simpler to use, and written in a more resilient way that crashed less. I can cut lnav some slack for the crashes because identifying and parsing arbitrary log formats seems like a messy problem. Still it shouldn't crash 1/3rd of the time I use it.

Sorry for the crashes :( I've been trying to improve it's internals more than adding features as of late. If you haven't already, please file bugs on github and/or submit crash logs to the mailing list. (I've taken a break from working on it lately. So, if you've done that and I haven't gotten back, I apologize.)

No worries. It's a great tool and I'd be very happy to make reports and share crash logs.

Re: Tips for analyzing logs

#130

Post by psd1 »

digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

> cat

> pet peeve

nice

Post reply on HN