Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tips for analyzing logs

jvns.ca

Post reply on HN
Print view View original

101–110 of 136 posts

Re: Tips for analyzing logs

#101

Post by foobarian »

kazinator wrote: post #94
digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

> It really demonstrates that you don't have the usage of the command receiving the output completely memorized. No, it demonstrates that you don't have redirection memorized, and don't know that you can place it anywhere in the command line, including on the left. > So you're wasting a few clock cycles Keystrokes too: cat x | cmd It's also possible that cmd may detect that its standard input is connected to a real f…

I still prefer to have cat there because it is interchangeable with other output-producing commands and it can handle globs. In an interactive session I iterate on the last command many times, and if I decide to filter stuff can just replace cat with grep or if I decide to pull from a directory of files can add a glob, if compressed it turns to zgrep or zcat etc. With redirects I'd have to change the structure of the pipeline which wastes mental effort. IMO.

Re: Tips for analyzing logs

#102

Post by edweis »

Best tips I discovered: use emojis. They have colors and they are easy to spot.

For instance if an API call is made use the phone emoji, when there is a timeout use a clock, when an order is dispatched used a package...

When you have to go through huge log file it is a life saver.

Re: Tips for analyzing logs

#103

Post by zug_zug »

I feel like this is a huge anti-pattern. Use a hosted service that does all of this for you, and then have a whole query language, build alerts, graphs, etc based on these results.

It's not super cheap, but it's 10x cheaper than wasting dev time in the terminal. (Sumologic, splunk are the two I can vouch for)

Re: Tips for analyzing logs

#104

Post by lowbloodsugar »

rsdbdr203 wrote: post #70
lowbloodsugar wrote: post #41

Earlier quoted context omitted.

And you should produce a single line of output for each request that identifies all of the pertinent information. You can have more than one, for e.g. a thread dump, but there should be one that provides a complete summary. I've lived with apps that logged at each stage of the process as separate lines, and that's just not useful data when grepping for anomalies.

I hypothesize it's not useful because you're using grep. If you use a tool that can show you multiple lines all tied by a request ID, it becomes much more helpful.

I am quite often looking for patterns across thousands of requests. As an example, one thing I inherited didn’t even log how long each request took to serve. Sure, you find out how long a single request took, by comparing the first and last log entry, but that’s just not useful 99% of the time.

Re: Tips for analyzing logs

#105

Post by spc476 »

One thing we did at my previous job was to add a "trace flag" to each account. Normally, they log nothing about a transaction (other than it happened), but if the trace flag is set, then a lot of information is logged about the transaction. Also, this trace flag is propagated throughout the distributed system they have, so we can trace the action across the network.

Re: Tips for analyzing logs

#106

Post by spc476 »

hayst4ck wrote: post #95

Some extra tips: Keep access logs, both when a service receives a request and finishes a request. Record request duration. Always rotate logs. Ingest logs into a central store if possible. Ingest exceptions into a central store if possible. Always use UTC everywhere in infra. Make sure all (semantic) lines in a log file contain a timestamp. Include thread ids if it makes sense to. It's useful to log unix timestamp al…

Some of these are KPIs (Key Performance Indicators). What we did at a previous job was to have a system like Etsy's statd [1] (it's an easy system to implement) and it made it easy to add statistics like latency of requests, number of errors, just about anything that could be measured, without excessive overhead (in terms of source code).

[1] https://www.datadoghq.com/blog/statsd/

Re: Tips for analyzing logs

#107

Post by joshka »

oefrha wrote: post #19
heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Huh, I almost posted a duplicate recommendation. My only complaint with lnav was that it had to be built from source on Linux and the build was frigging huge. Apparently they have a pre-compiled linux-musl binary now.

I used nix when I had to make lnav work on some machines where this would have been difficult.

Re: Tips for analyzing logs

#108

Post by joshka »

heliostatic wrote: post #2

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Came here to say the same - massive props to the lnav devs.

Re: Tips for analyzing logs

#109

Post by linsomniac »

I recently used clickhouse-local to do some log analysis on a lot of elastic load balancer logs (~10s of GBs) and it was spectacular.

In short, you can add clickhouse-local to a shell pipeline and then run SQL queries on the data. An example from the docs:

$ ps aux | tail -n +2 | awk '{ printf("%s\t%s\n", $1, $4) }' | clickhouse-local --structure "user String, mem Float64" --query "SELECT user, round(sum(mem), 2) as memTotal FROM table GROUP BY user ORDER BY memTotal DESC FORMAT Pretty"

Re: Tips for analyzing logs

#110

Post by DesiLurker »

in addition to these one of my favorite is a perl one liner that generates time delta from a regex pattern of interest. And then I plot it using gnuplot. It seriously helps to 'see' the events with timing in a chart & allows you to do a quick visual search for problem areas.
Post reply on HN