If you are using Kubernetes, I highly recommend using https://github.com/stern/stern
I second that. It's fast and conveniant, and you can visualize logs accross several instances easily,
Tips for analyzing logs
81–90 of 136 posts
Re: Tips for analyzing logs
#82One thing I've done to identify infrequent log entries within a log file is to remove all numbers from a file and print out a frequency of each. Basically just helps to disregard timestamps (not just at the beginning of the line), line numbers, etc. cat file.log | sed 's/[0-9]//g' | sort | uniq -c | sort -nr This has been incredibly helpful in quickly resolving outages more than once.
It's probably worth exploring making that a frequency bucketing pipe that dashboards N minute intervals .. so that operators can see any abrupt changes.
Re: Tips for analyzing logs
#83As much as I approve of a skillset to analyze local logs, but after a relatively small scale (10-20 systems), a central decent log aggregation like opensearch or ELK just brings so much value even on 1-3 nodes. It'd be one of the first changes I make to an infrastructure because it's so powerful. And its not just log searching and correlation value. At work, the entire discussion "oh but we need access to all servers…
syslog has allowed forwarding logs long before elk
Re: Tips for analyzing logs
#84One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…
I don't often (err, ever…) reply without reading further but this time I must, because: I've never heard this turn of phrase "useless use of cat" and it turned my brain upside-down for a moment, because: "interactive" is precisely how I learn and do and I suppose it was a nice reminder that sometimes really big (read: useless) things are actually kinda small (useful) and vice versa.
Re: Tips for analyzing logs
#85journalctl supports grep, queries by priority, time, daemon, and custom defined fields.
Re: Tips for analyzing logs
#86One thing I've done to identify infrequent log entries within a log file is to remove all numbers from a file and print out a frequency of each. Basically just helps to disregard timestamps (not just at the beginning of the line), line numbers, etc. cat file.log | sed 's/[0-9]//g' | sort | uniq -c | sort -nr This has been incredibly helpful in quickly resolving outages more than once.
Brilliant hack. I've used just about all the tricks from the blog and many of the comments here, but never this one. I've stripped timestamps for sure, but never considered all numerics. Nice one !
Re: Tips for analyzing logs
#87My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.
I love lnav and use it constantly, but it crashes a lot. I do wish there was something like lnav that was a little simpler to use, and written in a more resilient way that crashed less. I can cut lnav some slack for the crashes because identifying and parsing arbitrary log formats seems like a messy problem. Still it shouldn't crash 1/3rd of the time I use it.
Sorry for the crashes :( I've been trying to improve it's internals more than adding features as of late. If you haven't already, please file bugs on github and/or submit crash logs to the mailing list. (I've taken a break from working on it lately. So, if you've done that and I haven't gotten back, I apologize.)
Re: Tips for analyzing logs
#88“scroll through the log really fast”
This is a great point and probably underrated.
You may also benefit by scrolling sideways really fast.
Re: Tips for analyzing logs
#89My tips: 1) Fuck grep, use ripgrep, especially if you have to scour over an entire directory. 2) Get good with regex, seriously, it will shave hours off your searching. 3) For whatever application you are using, get to know how the logging is created. Find the methods used where said logs are made, and understand why such a log line exists. 4) Get good with piping into awk if needed if you need some nice readable out…
I use ack for this purpose, because the color filtering options are so good.
Re: Tips for analyzing logs
#90One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…
> Sorry. I did say, it is a pet peeve.
You're not alone. I disable the warning in shellcheck. I've been uselessly cat'ing since the 90s, like you.