Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Skip grep, use awk

blog.jpalardy.com

Post reply on HN
Print view View original

121–130 of 136 posts

Re: Skip grep, use awk

#121

Post by omaranto »

jedisct1 wrote: post #120
vidarh wrote: post #96

Earlier quoted context omitted.

Here's the equivalent for Ruby: alias glorp='ruby -ane ' $ [data is generated] | glorp ' ~ /Something/ and puts $F[2]' Or: $ echo -e '{"hello":"world"}\n{"hello":"cat"}' | glorp 'puts JSON.load($_)["hello"] ' -rjson (Of course Ruby got the -a autosplit-mode and the -n assumed 'while gets(); ... end' loop from Perl along with $_ and $F, so it's very intentional that they're similar)

And Ruby regexes are amazing.

I thought that they were amazing because they were just like Perl's. Are there any differences?

Re: Skip grep, use awk

#123

Post by fiddlerwoaroof »

derriz wrote: post #97
joosters wrote: post #73

Earlier quoted context omitted.

I wish the default 'cut' implementation could be just a little more clever - regex delimiters would be good, it doesn't even support multiple characters :( Also, cut's output manipulation is surprising. '-f 2,1' is actually the same as '-f 1,2' - you can't change the order of printing. I know there are other programs that can do the job, but it's a little frustrating when you can 'almost' get there with a chain of pi…

If you pipe your file through "while read f1 f2 f3 ; do echo field2 is $f2 ; done" for example you can pick out fields. Re-ordering them is just of a special case of any sort of bash manipulation you can do in the loop body. Admittedly for the very basic case, it's not as terse as "cut -f2" but if you're doing any further processing on the stream then I find it's often shorter.

Yeah, I thought of mentioning this. The reason I don't do this, though, is that I generally dislike keeping track of when the shell does or does not automatically split input, so I only use `read` with the -r option.

Re: Skip grep, use awk

#124

Post by racl101 »

I've always wanted to learn to use awk but I just can never find enough examples that allow me to understand what the hell I'm doing. The learning curve is too high for me.

Re: Skip grep, use awk

#126

Post by sebcat »

aargh_aargh wrote: post #113
avaika wrote: post #93

Earlier quoted context omitted.

why do you need `while read` part at all? this is the main slow down here. shell is extremely time consuming compared to sed/awk/whatever tool.

The script is currently a filter - the input comes from stdin.

You can redirect stdin to sed directly, e.g.,

    cat
or,

    cat  myfile
    A Brave Cow
    Was Walking
    Crows Were Airborne
    All Was Well
    EOF
    sed -e 's/A/X/;s/B/Y/;s/C/Z/;'
or (copy-pasted from shell for clarity),

    $ cat foo.sh
    #!/bin/sh
    sed -e 's/A/X/;s/B/Y/;s/C/Z/;'
    $ cat

Re: Skip grep, use awk

#127

Post by leovonl »

Ultimatt wrote: post #88

Skip awk, use perl.... The alias below sets perl to loop over STDIN splitting each line on more than one whitespace character and populate an array F. The -nE will then Evaluate an expression from the command line looping over the input line-by-line. alias glorp='perl -aF"/\s+/" -nE' So now we have the command `glorp` to play with which has more familiar syntax than awk and all of CPAN available to play with! $ [data…

Sorry, but I strongly disagree.

I can think of two very strong reasons to prefer awk over perl:

* busybox implements awk, so its (usually) available even on embedded platforms

* awk syntax is way clearer than perl, anyone that has a few experience in C + shell can figure out what is being done with some googling

awk is also powerful enough to separate output to different files, accumulate inputs, etc. And if you need anything more complex, there are tons of other languages to choose from then (including perl).

Re: Skip grep, use awk

#128

Post by aargh_aargh »

sebcat wrote: post #126
aargh_aargh wrote: post #113

Earlier quoted context omitted.

The script is currently a filter - the input comes from stdin.

You can redirect stdin to sed directly, e.g., cat or, cat myfile A Brave Cow Was Walking Crows Were Airborne All Was Well EOF sed -e 's/A/X/;s/B/Y/;s/C/Z/;' or (copy-pasted from shell for clarity), $ cat foo.sh #!/bin/sh sed -e 's/A/X/;s/B/Y/;s/C/Z/;' $ cat

Thank you, that helped. I had to get rid of '^' anchors (because it's not processing individual lines anymore) and I put

  sed -e ''
into the script, but the solution is now much simpler and faster.

Re: Skip grep, use awk

#129

Post by Ultimatt »

codesnik wrote: post #109
Ultimatt wrote: post #88

Skip awk, use perl.... The alias below sets perl to loop over STDIN splitting each line on more than one whitespace character and populate an array F. The -nE will then Evaluate an expression from the command line looping over the input line-by-line. alias glorp='perl -aF"/\s+/" -nE' So now we have the command `glorp` to play with which has more familiar syntax than awk and all of CPAN available to play with! $ [data…

what's the reason of using -F option? -a defaults " " separator, which already emulates awk behavior. If I'm not wrong, only difference between perl -aF"/\s+/" and perl -a is treating of leading whitespace in lines

To show you could use other separators defined by any regex you like.

Re: Skip grep, use awk

#130

Post by bsg75 »

tannhaeuser wrote: post #117

I'm sure the author is aware, but awk has at least three implementations: nawk (the one true), gawk (what most are using), and mawk (performance-oriented, unmaintained). Plus busybox-awk. When benchmarking gawk, I've found using LANG=C and avoiding UTF-8 to make a substantial difference for pattern matching.

Mawk is currently maintained: http://invisible-island.net/mawk/mawk.html
Post reply on HN