Stop using tail -f (mostly) • Hacker News Board

Re: Stop using tail -f (mostly)

#101

Post by shawabawa3 » Mon, Mar 30, 2015, 7:17 PM UTC

teraflop wrote: post #91

lgas wrote: post #89

Earlier quoted context omitted.

This has nothing to do with what is being discussed in this subthread.

Sure it does. If your log lines are distinct (e.g. they have timestamps or unique IDs) then you can use less's search highlighting to provide a visual marker for a specific line, similar to what you can do by manually inserting a bunch of blank lines on the console. This trick doesn't work if your log file has a bunch of identical lines and you want to keep an eye on their rate, though.

imagine the scenario: "I want to see everything that happens in a single request"

Enter method:

  1. Press enter a bunch of times
  2. Reload browser
  3. Press enter a bunch of times and scroll up

Your method:

  1. Search for last line in output to highlight it?
  2. Reload page
  3. Try and figure out where stuff starts and ends with loads of visual noise

Re: Stop using tail -f (mostly)

#102

Post by w4tson » Mon, Mar 30, 2015, 7:18 PM UTC

+1 for Commandline Fu. I love it when you see a colleague at a terminal and you say "Whoah, go back, what was that you just did?". It's kinda beautiful really, the way everyone has slightly different methods for common dev tasks at the commandline.

My method is usually '+G' to go to the end of the log file and check the damage report. Quick reverse search with '?' + search term to find last error, then probably hit 'n' to find the next previous any other occurrences to determine how frequent it's popping up. If it looks like it's a recurring problem i'll '+F' to see follow latest output. Then CTRL+c to quit that mode.

tail -f still has it's uses for cases as pimlottc mentioned. I like to put a bit of distance between the last bit of output. Especially if its a particularly repetitive and lengthy stacktrace.

Re: Stop using tail -f (mostly)

#103

Post by amelius » Mon, Mar 30, 2015, 7:20 PM UTC

Why not just pipe the output of tail -f through less?

Re: Stop using tail -f (mostly)

#104

Post by EdiX » Mon, Mar 30, 2015, 7:23 PM UTC

vbezhenar wrote: post #73

pjungwir wrote: post #72

Earlier quoted context omitted.

Another less-vs-vi question: As a junior programmer I got chewed out by my IT department because I was examining a (production) log file in vi. The sysadmin told me I should use less (actually more---this was on Solaris in ~2001). To this day I only read log files with less, but I've never figured out his objection. Negatives to using vi I can imagine are: - I might write to the log file. That seems like a real worry…

On large files less fires up immediately and vim is quite slow. I think that vim is loading an entire file in the memory while less just loads visible chunk. May be other vi implementations are more efficient with large files. Besides — vim really shouldn't be used on log files. It's editor, not viewer.

> I think that vim is loading an entire file in the memory while less just loads visible chunk

No, vim also only keeps part of the file loaded, however it will try to count how many lines the file has.

Re: Stop using tail -f (mostly)

#105

Post by TallGuyShort » Mon, Mar 30, 2015, 7:26 PM UTC

laumars wrote: post #100

> We all have been there: You are watching a file with tail -f, and then you need to search for something in this file, or just navigate up and down. Now you need to exit tail (or open a new shell), and ack this file or open it with vim to find what you are looking for. After that, you run tail again to continue watching the file. There's no need to do that when you are using less. There's also no need to do that if…

Not if you also need to search things that weren't at the end of the file when you started tailing - which is very common with log files.

Re: Stop using tail -f (mostly)

#106

Post by cromo » Mon, Mar 30, 2015, 7:29 PM UTC

drzaiusapelord wrote: post #13

tikums wrote: post #6

Or use multitail. It supports syntax highlighting, among other features: http://www.vanheusden.com/multitail/

multitail is great. I really cannot use command line utils that don't support colors/highlighting anymore. htop instead of top, etc. It just gives me much better visibility and readability. I'm surprised that so many non-color utils are still being used. It just feels so 1994 to me to stare at a white on black display. I guess there's nothing more slow moving and conservative than shell interfaces, thus articles like…

I wrote a Python script (called synesthesia)[1] that'll colorize input based on regex matches, and any matched text will be the same color for the same content.

The use case that drove its development was needing to keep track of UUIDs across multiple logs - and grep --color will colorize its matches, but not differentiate between ones that have different content. With this, I could watch both logs as data was passed from one to the other and keep track of e.g. the orange one.

I also thought it would be nice to be able to use patterns from logstash's grok, so I wrote grokpat[2] to find patterns for me. A lot of grok's patterns use atomic groups, which aren't available in Python 2, so I wrote redi[3] to convert them from grok's syntax to Python compatible syntax.

So I can now colorize logs easily as follows:

  tail -F program.log | synesthesia "$(redi "$(grokpat uuid)")"

[1] https://github.com/cromo/synesthesia [2] https://github.com/cromo/grokpat [3] https://github.com/cromo/redi

Re: Stop using tail -f (mostly)

#107

Post by gojomo » Mon, Mar 30, 2015, 7:33 PM UTC

teraflop wrote: post #91

lgas wrote: post #89

Earlier quoted context omitted.

This has nothing to do with what is being discussed in this subthread.

Sure it does. If your log lines are distinct (e.g. they have timestamps or unique IDs) then you can use less's search highlighting to provide a visual marker for a specific line, similar to what you can do by manually inserting a bunch of blank lines on the console. This trick doesn't work if your log file has a bunch of identical lines and you want to keep an eye on their rate, though.

Having to remember & type a timestamp has much more mental overhead (planning & memory) than "scan/scroll back to last block of vertical whitespace".

That's why suggestions of either named-marks or back-searches aren't considered equally-attractive alternatives to marking the scrollback with a batch of s.

Re: Stop using tail -f (mostly)

#108

Post by nkouevda » Mon, Mar 30, 2015, 7:35 PM UTC

gitaarik wrote: post #50

jfroma wrote: post #48

Earlier quoted context omitted.

This. I use: tail -f /var/log/x.log | grep foo -A 20 -B 20 Most of the time.

You know you can also do -C 40 for 40 lines of context instead of -A 20 -B 20

Note that -C is in each direction, not total; -C20 is equivalent to -A20 -B20.

Re: Stop using tail -f (mostly)

#109

Post by ekimekim » Mon, Mar 30, 2015, 7:35 PM UTC

haberman wrote: post #94

spatz wrote: post #27

The problem is that less +F polls every second while tail -f uses inotify, which is both more efficient and responds faster to change.

I always thought it was such a shame that you couldn't use plain poll() for this. poll()-ing on a read fd at EOF for a regular file should work like poll()-ing a network socket.

The problem is in what concepts are being considered "the same" when mapping two mostly-dissimilar things (a socket or pipe, vs a file).

poll(), select() et al don't define "readable" as meaning "a byte of data is available". They define it as "read() will not block".

When you read to the end of a file, read() returns "". This is the same as when you try to read from a closed socket. Conceptually, they are linking the concepts of "EOF" and "closed", not the concepts of "EOF" and "waiting for data". And indeed, if you call poll() on a socket that has been closed on the remote end, you will find it is "readable".

Ultimately, regular files just were never intended to be used as pipes. The abstractions just weren't chosen to work in that way.

Re: Stop using tail -f (mostly)

#110

Post by technofiend » Mon, Mar 30, 2015, 7:37 PM UTC

bch wrote: post #83

pjungwir wrote: post #70

Earlier quoted context omitted.

Just the other day I was looking for a way to do this in less: yank 10 lines, open a new file, paste the 10 lines, save the file. Is that possible? My Googling last week didn't find a good solution.

If you wanted to grab 10 lines from /var/log/messages, starting at the first line that has "pjungwlr" in it, you could: $ less /var/log/msgs /pjungwlr^M ^G (note line informational line numbers) v (launch vi) [double check your line #s, etc] :d1,. (delete from 1st line, to current line) 10j (go down 10 lines) :d,$ (delete to EOF) :w my_newfile :q

This is such a common problem there's a stackoverflow article on it.

http://stackoverflow.com/questions/17908555/printing-with-se...

(Actually, there's probably more than one despite their really good duplication grooming, but this was the first hit.)