Live data from Hacker News

Skip grep, use awk

blog.jpalardy.com

131–136 of 136 posts

Re: Skip grep, use awk

#131
post #91

Earlier quoted context omitted.

Hey, it would be awesome to have something like this for Python!

I wrote a little tool called "pyline" about a hundred years ago... I still reach for it often when I'm in a hurry, and don't have time to futz around with awk/sed. https://code.activestate.com/recipes/437932-pyline-a-grep-li...

That's pretty cool, thanks!

Re: Skip grep, use awk

#132
post #6

While awk is indeed under-appreciated, there are many instances where using grep to pre-filter your input is helpful because the grep family (GNU grep, agrep, etc.) can match strings much much faster than awk's regex engine. For example: GNU grep uses a heavily optimized implementation of the Boyer-Moore string search algorithm [1] which (it is claimed) requires just 3 (x86) cycles per byte of input. Boyer-Moore only…

I have found ack and ag to be much faster than grep (I should actually time this).

Creator of ack here.

The three biggest reasons to use grepalikes is not necessarily the time to execute the search, but

1) The reduced amount of typing. It's much faster to type "ack foo --ruby" than it is to type "find . -name '* .rb' | xargs grep foo"

2) Better filetype matching, because ack and ag check things beyond just file extensions. ack's --ruby flag checks for *.rb and Rakefile and "ruby" appearing in the shebang line, for example.

3) More features that grep doesn't include, like better highlighting and grouping, or full Perl regular expressions (in ack) and a ton of other features.

Here's a command line to find all the header files in your C source. It takes advantage of the ability to use Perl's regular expressions to specify output:

ack --cc '#include ' -H --output='$1' | sort -u

Here's an article I wrote a while ago on comparing the features of ack and ag. https://blog.newrelic.com/2015/01/28/grep-ack-ag/

Even though I created ack, I don't care which tool you use, so long as it suits your needs and makes you happy, but if all you're using your grep/ack/ag for is "grep -R function_name ." then you're only scratching the surface of what the tools can do for you.

Re: Skip grep, use awk

#133

Earlier quoted context omitted.

I have found ack and ag to be much faster than grep (I should actually time this).

They are perceived as faster because they automatically skip binary and VCS-ignored files. grep is faster.

> they automatically skip binary and VCS-ignored files

If it does less work, and takes less time, then isn't that faster?

Re: Skip grep, use awk

#134
post #126

Earlier quoted context omitted.

You can redirect stdin to sed directly, e.g., cat or, cat myfile A Brave Cow Was Walking Crows Were Airborne All Was Well EOF sed -e 's/A/X/;s/B/Y/;s/C/Z/;' or (copy-pasted from shell for clarity), $ cat foo.sh #!/bin/sh sed -e 's/A/X/;s/B/Y/;s/C/Z/;' $ cat

Thank you, that helped. I had to get rid of '^' anchors (because it's not processing individual lines anymore) and I put sed -e '' into the script, but the solution is now much simpler and faster.

sed works on stdin by default (see third example) so there shouldn't be a need to pipe from /dev/stdin, and line anchors should still work.

    $ cat myfile 
    foo foo
    baz baz
    $ cat foo.sh
    #!/bin/sh
    sed 's/^foo/bar/;s/^baz/foobar/'
    $ ./foo.sh 
Depending on your expression, you may want to use -E to get POSIX ERE syntax.

Re: Skip grep, use awk

#136
post #99

Earlier quoted context omitted.

There is "Pyed Piper" aka `pyp` https://code.google.com/archive/p/pyp/ "Pyp is a linux command line text manipulation tool similar to awk or sed, but which uses standard python string and list methods as well as custom functions evolved to generate fast results in an intense production environment."

That's cool, but I feel like you're unlikely to run into it very often in the wild? The vast majority of Linux systems come with ruby/perl out of the box. Plus if Python did support this sort of hackery it would quickly garner most of the bad press Perl has over the years :'(

Who cares? It's on pypi (https://pypi.python.org/pypi/pyp/2.11) so just `pip install pyp` and then you have it. You're probably not going to want to distribute shell scripts that rely on pyp, but if you're distributing scripts you can just distribute scripts written entirely in Python and use pyp for your own one-liners.
Post reply on HN