Live data from Hacker News

Skip grep, use awk

blog.jpalardy.com

31–40 of 136 posts

Re: Skip grep, use awk

#31

My favourite bad example of using grep was from a big enterprise software vendor to kill one of their processes. It looked something like ps -ef| grep SomeDaemon | grep -v grep | grep -v perl | perl -e ' '

Serious question: what would be a better way to do this?

I would just recommend pkill: https://en.wikipedia.org/wiki/Pkill

Re: Skip grep, use awk

#32

My favourite bad example of using grep was from a big enterprise software vendor to kill one of their processes. It looked something like ps -ef| grep SomeDaemon | grep -v grep | grep -v perl | perl -e ' '

Serious question: what would be a better way to do this?

PowerShell?

Re: Skip grep, use awk

#33

Earlier quoted context omitted.

Serious question: what would be a better way to do this?

I would just recommend pkill: https://en.wikipedia.org/wiki/Pkill

I tend to use the following command:

    kill -HUP $(pidof SomeDaemon)
and if it insists on running, I use:

    sudo kill -9 $(pidof SomeDaemon)
That's it, really.

Re: Skip grep, use awk

#34
But I like my colorized match: grep --color ...

Also grep -n is so much nicer than something like awk '{print NR "," $0}'

Finally, I think grep must be faster if only because grep doesn't line buffer by default, although you may --line-buffered.

Re: Skip grep, use awk

#35
post #11

Earlier quoted context omitted.

See http://blog.burntsushi.net/ripgrep/ for a quite nice comparison which is counter to your experience; ack/ag/pt are all slower than either grep or ripgrep.

"ack/ag/pt are all slower than either grep or ripgrep" You need to be a touch careful with blanket statements, as this conclusion isn't quite what the data in my blog says. What it says is that ag is generally slower than GNU grep on large files, primarily because GNU grep's core search code is quite a bit more optimized than ag's. However, ag can outclass GNU grep when searching across entire directories primarily b…

Regarding your '(foo){100}' example, couldn't you expand this into a literal search for 100 repetitions of 'foo'? I guess this could interact poorly with the regex engine, and you'd be expected to cover more complex instances like '(foo){50,100}', but I think it might be worth the effort in some cases.

Re: Skip grep, use awk

#36
post #15

Earlier quoted context omitted.

They are perceived as faster because they automatically skip binary and VCS-ignored files. grep is faster.

They are faster for practical usage, though I'm fairly sure ag also executes the search in parallel no? I also seem to recall the ripgrep author recently trying to optimise towards grep in another HN post. Love ag ever since I discovered it, though you have to be careful every so often it doesn't look in a filetype that mattered.

[deleted]

Re: Skip grep, use awk

#37

My favourite bad example of using grep was from a big enterprise software vendor to kill one of their processes. It looked something like ps -ef| grep SomeDaemon | grep -v grep | grep -v perl | perl -e ' '

Serious question: what would be a better way to do this?

[deleted]

Re: Skip grep, use awk

#38

Earlier quoted context omitted.

I would just recommend pkill: https://en.wikipedia.org/wiki/Pkill

I tend to use the following command: kill -HUP $(pidof SomeDaemon) and if it insists on running, I use: sudo kill -9 $(pidof SomeDaemon) That's it, really.

I suggest:

    killall -HUP SomeDaemon

Re: Skip grep, use awk

#39
post #35

Earlier quoted context omitted.

"ack/ag/pt are all slower than either grep or ripgrep" You need to be a touch careful with blanket statements, as this conclusion isn't quite what the data in my blog says. What it says is that ag is generally slower than GNU grep on large files, primarily because GNU grep's core search code is quite a bit more optimized than ag's. However, ag can outclass GNU grep when searching across entire directories primarily b…

Regarding your '(foo){100}' example, couldn't you expand this into a literal search for 100 repetitions of 'foo'? I guess this could interact poorly with the regex engine, and you'd be expected to cover more complex instances like '(foo){50,100}', but I think it might be worth the effort in some cases.

Yes, my example was bad. Consider `\pL{100}` instead. :-)

With respect to expanding repetitions on literals, that already works today. e.g.,

    $ rg '(foo){100}' /dev/null --debug
    ...
    DEBUG:grep::literals: required literal found: "foofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoo"
The truly eagle-eyed would notice that there are actually only 83 instances of `foo` in this string. In fact, literal detection is actually quite a tricky task on arbitrary regular expressions, and one must be careful to bound the number of and size of literals you produce. For example, the regex `\w+` matches an ~infinite number of literals, but `\w` is really just a normal character class, and some character classes are good to include in your literal analysis.

The 83 comes from the fact that 83 times 3 = 249, which buts up against a hard-coded limit in the regex engine for literal detection: https://github.com/rust-lang/regex/blob/d894c631cb6c9a062c13...

In this case, the literal detector knows that the literal is a prefix. That means a prefix match must still be confirmed by entering the regex engine. If you pass a smaller literal, e.g., `(foo){4}`, then you'll see slightly different output:

    DEBUG:grep::literals: literal prefixes detected: Literals { lits: [Complete(foofoofoofoo)], limit_size: 250, limit_class: 10 }
In this case, the regex compiler sees that a literal match corresponds to an overall match of the regex, and can therefore stay out of the regex engine entirely. The "completion" analysis doesn't stop at simple regexes either, for example, `(foo){2}|(ba[rz]){2}` yields:

    DEBUG:grep::literals: literal prefixes detected: Literals { lits: [Complete(foofoo), Complete(barbar), Complete(bazbar), Complete(barbaz), Complete(bazbaz)], limit_size: 250, limit_class: 10 }
This is important, because this particular pattern will probably wind up using a special SIMD multi-pattern matcher. Failing that, it will use the "advanced" version of Aho-Corasick, which is a DFA that is computed ahead-of-time without extra failure transitions (as opposed to the more general lazy DFA used by the regex engine).

Literal detection is a big part of the secret sauce of ripgrep. Pretty much every single regex you feed to a tool like ripgrep will contain a literal somewhere, and this will greatly increase search speed when compared to tools that don't do literal extraction.

Of course, literal extraction has downsides. If your literal extractor picks out a prefix that is a very very common string in your haystack, then it would probably be better to just use the regex engine instead of ping-ponging back-and-forth between the prefix searcher and the regex engine. Of course, there are probably ways to detect and stop this ping-ponging, but I haven't invested much effort in that yet. Alas, the performance improvement in the common case (literals make things faster) seems to wind up being more beneficial for the general use case.

There is more in my blog post: http://blog.burntsushi.net/ripgrep/#literal-optimizations

Re: Skip grep, use awk

#40

My favourite bad example of using grep was from a big enterprise software vendor to kill one of their processes. It looked something like ps -ef| grep SomeDaemon | grep -v grep | grep -v perl | perl -e ' '

Serious question: what would be a better way to do this?

As per other users, you can use pkill but it's not entirely portable in meaning between OSs. This software had to run on just about every possible Unix out there. Solaris, HP-UX, AIX, many others ...

My main point is that using perl and grep, (multiple thereof!) is nuts when you can do it all in perl.

Also crazy was using -ef (all processes) when the process of interest was using a known user. So ps -u would be more appropriate.

Doing as much in Perl as possible, for portability, you'd do something like

  ps -u | perl -ane 'm/[S]omeDaemon/ && kill "SIGTERM", $F[1]'
-a means autosplit into fields, $F[0], F[1], etc so the pid is $F[1].

See `perldoc -f kill` for the Perl function kill.

You can also do the `ps` from within Perl but I don't think you'd be gaining much in readability or portability.

Post reply on HN