If I had a nickel for every time I have used this pattern
... | perl -ne 'print "$1 $2 ...\n" if /some-regex-with-capture-groups/' | ...
Tips for analyzing logs
131–136 of 136 posts
Re: Tips for analyzing logs
#132As much as I approve of a skillset to analyze local logs, but after a relatively small scale (10-20 systems), a central decent log aggregation like opensearch or ELK just brings so much value even on 1-3 nodes. It'd be one of the first changes I make to an infrastructure because it's so powerful. And its not just log searching and correlation value. At work, the entire discussion "oh but we need access to all servers…
We went to the trouble of setting up ELK and I was excited, but I never use it any more. You can't scroll/search through the logs as fast as vi and grep. You have to click to see more log lines. And I can write an alias to tail a log so I don't have to log into a UI.
Re: Tips for analyzing logs
#133One of my favorite tricks is to use a visual difftool. Copy good log into left panel. Copy bad log into right panel. Quickly show which lines are new, which are missing and which are out of order. Obviously ignore the timestamps ;)
I do this all the time (PyCharm has an amazing diff tool hidden under Cmd-Shift-A, Show Diff). With the command from above that removes all the numeric characters it's going to be even easier to compare, no more timestamps!
Re: Tips for analyzing logs
#134One thing I've done to identify infrequent log entries within a log file is to remove all numbers from a file and print out a frequency of each. Basically just helps to disregard timestamps (not just at the beginning of the line), line numbers, etc. cat file.log | sed 's/[0-9]//g' | sort | uniq -c | sort -nr This has been incredibly helpful in quickly resolving outages more than once.
I love this tip for all kinds of diffing. I'm so sure I'm going to use it that I already assigned it an alias to strip numbers from whatever's on my Mac clipboard:
alias numberless="pbpaste | sed 's/[0-9]//g' | pbcopy"Re: Tips for analyzing logs
#135Best tips I discovered: use emojis. They have colors and they are easy to spot. For instance if an API call is made use the phone emoji, when there is a timeout use a clock, when an order is dispatched used a package... When you have to go through huge log file it is a life saver.
This seems like a prime candidate for a "7 tips for app devs SREs will hate" article, that only works because you're the only one doing it.
Re: Tips for analyzing logs
#136Loosely related: a few years ago I wanted a simpler alternative to some of the more feature-full log viewers out there so I threw together a tiny (50kb) app that might be useful to some folks in here. All it does is consistently colors the first field in a line from stdin so you can quickly see which log lines have the same first field. I used it in combination with the parallel[0] command to prefix log lines by repl…
Nice tool. You should really consider including a screenshot showing the actual coloured output on the GitHub README.