Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tips for analyzing logs

jvns.ca

Post reply on HN
Print view View original

41–50 of 136 posts

Re: Tips for analyzing logs

#41

Post by lowbloodsugar »

pantulis wrote: post #11
kerblang wrote: post #5

> you’ll get overwhelmed by a million irrelevant messages because the log level is set to INFO I know this happens, but I think it's because programmers are abusing INFO. In principle it's reserved for messages that are informative at a level sys admins and a few others can make sense of and use. Unfortunately abuse often leads to "We turned INFO off" making it much harder to diagnose things after the fact.

I think there should be an counterpart to "log analysis" which is "logging strategies for your app". Which should be WHAT to log and WHEN. Stuff like, if you are exposing an HTTP endpoint, you should log the request URL and the time it took to serve it. Or if you are invoking an extenral service, you should log the response time of that service.

And you should produce a single line of output for each request that identifies all of the pertinent information. You can have more than one, for e.g. a thread dump, but there should be one that provides a complete summary. I've lived with apps that logged at each stage of the process as separate lines, and that's just not useful data when grepping for anomalies.

Re: Tips for analyzing logs

#42

Post by 10000truths »

jldugger wrote: post #36

Honestly, the most amazing thing I did with logs was learn how to do subtraction. Any time you have multiple instances of a thing and only some of them are bad, you can easily find the problem (if anyone bothered to log it) by performing bad - good. The way you do this is by aggregating logs by fingerprints. Removing everything but punctuation is a generic approach to fingerprinting, but is not exactly human friendly…

Isn’t this basically what structured binary logs are? Instead of writing a string like `Error: foo timeout after ${elapsed_amt} ms` to the log, you write a 4-byte error code and a 4 byte integer for elapsed_amt. I know there are libraries like C++’s nanolog that do this for you, under the hood.

Re: Tips for analyzing logs

#43

Post by jldugger »

nerdponx wrote: post #39
jldugger wrote: post #36

Honestly, the most amazing thing I did with logs was learn how to do subtraction. Any time you have multiple instances of a thing and only some of them are bad, you can easily find the problem (if anyone bothered to log it) by performing bad - good. The way you do this is by aggregating logs by fingerprints. Removing everything but punctuation is a generic approach to fingerprinting, but is not exactly human friendly…

You might be interested in the TF-IDF algorithm used in information retrieval and text classification.

Yes, this is pretty much TF-IDF for people too lazy to count the number of unique items in the corpus.

Since that number should be the same (or at least close!) in both good and bad datasets, I'm not sure the extra math matters much.

Re: Tips for analyzing logs

#44

Post by jldugger »

10000truths wrote: post #42
jldugger wrote: post #36

Honestly, the most amazing thing I did with logs was learn how to do subtraction. Any time you have multiple instances of a thing and only some of them are bad, you can easily find the problem (if anyone bothered to log it) by performing bad - good. The way you do this is by aggregating logs by fingerprints. Removing everything but punctuation is a generic approach to fingerprinting, but is not exactly human friendly…

Isn’t this basically what structured binary logs are? Instead of writing a string like `Error: foo timeout after ${elapsed_amt} ms` to the log, you write a 4-byte error code and a 4 byte integer for elapsed_amt. I know there are libraries like C++’s nanolog that do this for you, under the hood.

That solves the fingerprint, but you still need to count and score.

Re: Tips for analyzing logs

#45

Post by ranger207 »

the_arun wrote: post #34

Do we still use utilities like grep for searching logs? Are these when we cannot stream logs to tools like Splunk & Loggly and use their search services?

All of our logs are in Kibana, but sometimes I'll `kubectl logs pod > tmp; grep pattern tmp` because Kibana's search and filtering is often annoying. Actually, I'll usually open the logs in vim and use its search, which also get me the ability to eg delete every line not containing the pattern and then search only things matching the first pattern. I'm going to try lnav as mentioned in this thread but I've gotten by fine with judicious use of grep

Re: Tips for analyzing logs

#46

Post by aargh_aargh »

chasil wrote: post #16

A minor optimization is collapsing the grep -v, from this: cat file | grep -v THING1 | grep -v THING2 | grep -v THING3 | grep -v THING4 to this: egrep -v 'THING1|THING2|THING3|THING4' file That gets rid of the cat and three greps. Both POSIX and GNU encourage grep -E to be used in preference to egrep. A pcregrep utility also used to exist, if you want expansive perl-compatible regular expressions. This has been absor…

I once compared the speed of these two approaches, rather accidentally. I did output colorization by adding ANSII sequences. I thought, of course one process must be more efficient than a pipe of processes. After the rewrite, I was disappointed about the slowdown and reverted back to the pipe.

PS I checked back and I used sed rather than grep. I think the result would hold for grep but the morale is that you should verify rather than assume.

I have around 50 seds in the pipe, running in parallel (which is what makes it faster), it would have been a half of that when I tried the rewrite.

Re: Tips for analyzing logs

#48

Post by aargh_aargh »

tstack wrote: post #9
clumsysmurf wrote: post #4

Earlier quoted context omitted.

Interesting! Do you know how it compares to multitail? ( https://www.vanheusden.com/multitail/ ) They look very similar.

I don't think multitail really understands logs like lnav does, it's just following the last lines in the file. For example, if you try to follow multiple files in multitail like so: $ multitail /var/log/install.log -I /var/log/system.log You get a view with the tail from one file followed by the tail from the other, they are not collated by timestamp. In contrast, if you do the same thing in lnav: $ lnav /var/log/in…

I migrated from multitail to lnav. Turned out to be a no-brainer.

I second the above, just one pain point with multitail to add. I often page/search/filter in the scrollback buffer (I typoed "bugger" - Freudian slip?) and in multitail the scrollback is a separate window with a frame and everything, which is a pain (copying whole lines using mouse includes the frame, ugh). The filtering/searching being a separate pain.

One thing I used in multitail and not sure if I migrated wholly to lnav was log file syntax highlighting using regexes.

Re: Tips for analyzing logs

#49

Post by danjc »

We've added a log tailing feature into our product UI which also has a basic find/filter. It's been enormously useful for cases where something weird happens as you can immediately access the last few mins of logs.

Re: Tips for analyzing logs

#50

Post by tetha »

As much as I approve of a skillset to analyze local logs, but after a relatively small scale (10-20 systems), a central decent log aggregation like opensearch or ELK just brings so much value even on 1-3 nodes. It'd be one of the first changes I make to an infrastructure because it's so powerful.

And its not just log searching and correlation value. At work, the entire discussion "oh but we need access to all servers because of logs" just died when all logs were accessible via one web interface. I added a log aggregation and suddenly only ops needed access to servers.

Designing that thing with accessibility and discoverability in mind is a whone nother topic though.

Post reply on HN