Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tips for analyzing logs

jvns.ca

Post reply on HN
Print view View original

91–100 of 136 posts

Re: Tips for analyzing logs

#91

Post by veyh »

janci wrote: post #8
Manjuuu wrote: post #7

> Often log lines will include a request ID. Yes, always include a request id in every request structure you create and include it also in the response and print it. It would seem something obvious that everyone does by default but instead, no, it's not so obvious it seems.

Not so obvious. How to implement it without passing request id to all the functions, when they are unrelated to request/http ? Especially in languages without thread-locals such as javascript?

For example: https://github.com/puzpuzpuz/cls-rtracer

Re: Tips for analyzing logs

#92

Post by chaps »

digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

It's a perfectly valid pet peeve, especially with things like `tr` being weird about accepting files as input.

This works:

  cat file.log | tr 'a' 'b'
This doesn't:

  tr 'a' 'b' file.log

Re: Tips for analyzing logs

#93

Post by kazinator »

chaps wrote: post #92
digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

It's a perfectly valid pet peeve, especially with things like `tr` being weird about accepting files as input. This works: cat file.log | tr 'a' 'b' This doesn't: tr 'a' 'b' file.log

   tr a b
The pattern "cat file | command" introduces an unnecessary cat process to achieve the effect of "It is not called "useless use of cat" (UUoC) without justification.

Re: Tips for analyzing logs

#94

Post by kazinator »

digitalsushi wrote: post #58

One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since. Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into. But, if you're doing a 'useless' use of…

> It really demonstrates that you don't have the usage of the command receiving the output completely memorized.

No, it demonstrates that you don't have redirection memorized, and don't know that you can place it anywhere in the command line, including on the left.

> So you're wasting a few clock cycles

Keystrokes too:

   cat x | cmd
It's also possible that cmd may detect that its standard input is connected to a real file, and take advantage of being able to lseek() the file descriptor. For instance say that x is an archive and cmd is an extractor. If cmd needs to skip some indicated offset to get to the desired payload material, it may be able to do it efficiently with an lseek, whereas under cat, it has to read all the bytes and discard them to get to the desired offset.

Re: Tips for analyzing logs

#95

Post by hayst4ck »

Some extra tips:

  Keep access logs, both when a service receives a request and finishes a request.
  Record request duration.
  Always rotate logs.
  Ingest logs into a central store if possible.
  Ingest exceptions into a central store if possible.
  Always use UTC everywhere in infra.
  Make sure all (semantic) lines in a log file contain a timestamp.
  Include thread ids if it makes sense to.
  It's useful to log unix timestamp alongside human readable time because it is trivially sortable.
  Use head/tail to test a command before running it on a large log file.
If you find yourself going to logs for time series data then it is definitely time to use a time series database. If you can't do that, at least write a `/private/stats` handler that displays in memory histograms/counters/gauges of relevant data.

Know the difference between stderr and stdout and how to manipulate them on the command line (2>/dev/null is invaluable, 2>&1 is useful), use them appropriately for script output.

Use atop, it makes debugging machine level/resource problems 10 fold easier.

Have a general knowledge of log files (sometimes /var/log/syslog will tell you exactly your problem, often in red colored text).

If you keep around a list of relevant hostnames:

  cat $hostname_list_file | xargs -P $parallelness -I XHOSTNAME ssh XHOSTNAME -- grep
This needs to be used carefully and deliberately. This is the style of command that can test your backups. This style command has caused multiple _major_ outages. With it, you can find a needle in a haystack across an entire fleet of machines quickly and trivially. If you need to do more complex things, `bash -c` can be the command sent to ssh.

I've had an unreasonable amount of success opening up log files in vim and using vim to explore and operate on them. You can do command line actions one at a time (:!$bash_cmd), and you can trivially undo (or redo) anything to the logs. Searching and sorting, line jumping, pagedown/up, etc, diffing, jump to top of file or bottom, status bar telling you how far you are into a file or how many lines it has without having to wc -l, etc.

Lastly, it's great to think of the command line in terms of map and reduce. `sed` is a mapping command, `grep` is a reducing command. Awk is frequently used for either mapping or reducing.

Re: Tips for analyzing logs

#96

Post by slyall »

the_arun wrote: post #34

Do we still use utilities like grep for searching logs? Are these when we cannot stream logs to tools like Splunk & Loggly and use their search services?

One problem is that you start using these services and then the bill arrives and you then spend all your time removing stuff from the logs to keep the bill down.

Re: Tips for analyzing logs

#97

Post by hayst4ck »

jpitz wrote: post #89
Severian wrote: post #23

My tips: 1) Fuck grep, use ripgrep, especially if you have to scour over an entire directory. 2) Get good with regex, seriously, it will shave hours off your searching. 3) For whatever application you are using, get to know how the logging is created. Find the methods used where said logs are made, and understand why such a log line exists. 4) Get good with piping into awk if needed if you need some nice readable out…

I use ack for this purpose, because the color filtering options are so good. https://beyondgrep.com/

Every time I've done research (twice) ripgrep comes out on top.

Usually it's something like rg > ag > ack > grep and that's overall testimonial, not just speed.

What am I missing?

Re: Tips for analyzing logs

#98

Post by easton »

hayst4ck wrote: post #95

Some extra tips: Keep access logs, both when a service receives a request and finishes a request. Record request duration. Always rotate logs. Ingest logs into a central store if possible. Ingest exceptions into a central store if possible. Always use UTC everywhere in infra. Make sure all (semantic) lines in a log file contain a timestamp. Include thread ids if it makes sense to. It's useful to log unix timestamp al…

> Always use UTC everywhere in infra

Can Amazon do this? They use UTC and your local browser’s time seemingly randomly depending on AWS service, and it drives me nuts. They usually (not always) put the timezone next to it, but why can’t they just have a mandate that it either is or is not UTC?! (The worst one is that the Lambda console is UTC but cloudwatch isn’t, so you think you haven’t received a request in hours but then you did)

Re: Tips for analyzing logs

#99

Post by chaps »

kazinator wrote: post #93
chaps wrote: post #92

Earlier quoted context omitted.

It's a perfectly valid pet peeve, especially with things like `tr` being weird about accepting files as input. This works: cat file.log | tr 'a' 'b' This doesn't: tr 'a' 'b' file.log

tr a b The pattern "cat file | command" introduces an unnecessary cat process to achieve the effect of " It is not called "useless use of cat" (UUoC) without justification.

It is not called "useless UUoC comment" (UUUoCC) without justification. ;)

From a personal taste perspective, I'm not a fan of either. Having a floating "But honestly, if it's such a big deal to have a cat process floating around, there are probably other things you should be concerned about. "Adds extra load to the server" points to other problems. If perf matters, CPU shielding should be used. Or if that's not an option, then sure, there's some room for trifling around, but if you're at a point where you're already running a series of pipes, a single cat command is beans compared to the greps and seds that come after it.

Re: Tips for analyzing logs

#100

Post by kjellsbells »

When I was training sysadmins back in the dark ages, one of the rules I taught was: know what good looks like in your logs. If you are scanning hundreds of lines of logging under duress to find a smoking gun, and you don't know the difference between what the logs normally show, and what you are seeing, you'll waste a lot of time.

Corollary is that good day logs should be minimal and "clean", e.g not logging a lot, or, logging nice and predictably (which makes them easy to strip out via grep -v, etc.)

Post reply on HN