Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Skip grep, use awk

blog.jpalardy.com

Post reply on HN
Print view View original

101–110 of 136 posts

Re: Skip grep, use awk

#101

Post by Ultimatt »

phaemon wrote: post #99
Ultimatt wrote: post #94

Earlier quoted context omitted.

Not really possible in Python for anything overly useful/complex because of the significant white space, and regular expressions are not a first class object in the language. ipython is the equivalent solution I suppose. However, Ruby completely inherited this aspect of Perl and is another good candidate. It's just even old systems have a perl installed that will perform the awk like functionality.

There is "Pyed Piper" aka `pyp` https://code.google.com/archive/p/pyp/ "Pyp is a linux command line text manipulation tool similar to awk or sed, but which uses standard python string and list methods as well as custom functions evolved to generate fast results in an intense production environment."

That's cool, but I feel like you're unlikely to run into it very often in the wild? The vast majority of Linux systems come with ruby/perl out of the box. Plus if Python did support this sort of hackery it would quickly garner most of the bad press Perl has over the years :'(

Re: Skip grep, use awk

#102

Post by dingo_bat »

beefsack wrote: post #17

ripgrep[1] is functionally incredibly similar to grep and ag, but is significantly faster[2] and supports a wider range of character encodings. In its short lifetime it has already become the default search tool for VSCode. I've switched to using it as my daily driver for text search and am incredibly happy with it. [1]: https://github.com/BurntSushi/ripgrep [2]: http://blog.burntsushi.net/ripgrep/ Edit: I confused a…

I tried a lot but could not install ripgrep on ubuntu :(

Apparently the code is in rust, which I know zero of. So, I didn't try building it either.

Re: Skip grep, use awk

#103

Post by vidarh »

derriz wrote: post #97
joosters wrote: post #73

Earlier quoted context omitted.

I wish the default 'cut' implementation could be just a little more clever - regex delimiters would be good, it doesn't even support multiple characters :( Also, cut's output manipulation is surprising. '-f 2,1' is actually the same as '-f 1,2' - you can't change the order of printing. I know there are other programs that can do the job, but it's a little frustrating when you can 'almost' get there with a chain of pi…

If you pipe your file through "while read f1 f2 f3 ; do echo field2 is $f2 ; done" for example you can pick out fields. Re-ordering them is just of a special case of any sort of bash manipulation you can do in the loop body. Admittedly for the very basic case, it's not as terse as "cut -f2" but if you're doing any further processing on the stream then I find it's often shorter.

In bash you can also do do "while read -a F ; do echo field2 is ${F:1} ; done". (-a will assign each field to an zero indexed array)

Re: Skip grep, use awk

#104

Post by burntsushi »

dingo_bat wrote: post #102
beefsack wrote: post #17

ripgrep[1] is functionally incredibly similar to grep and ag, but is significantly faster[2] and supports a wider range of character encodings. In its short lifetime it has already become the default search tool for VSCode. I've switched to using it as my daily driver for text search and am incredibly happy with it. [1]: https://github.com/BurntSushi/ripgrep [2]: http://blog.burntsushi.net/ripgrep/ Edit: I confused a…

I tried a lot but could not install ripgrep on ubuntu :( Apparently the code is in rust, which I know zero of. So, I didn't try building it either.

It's simple to install ripgrep on pretty much any Linux because I distribute statically compiled binaries:

    $ curl -LO 'https://github.com/BurntSushi/ripgrep/releases/download/0.5.2/ripgrep-0.5.2-x86_64-unknown-linux-musl.tar.gz'
    $ tar xf ripgrep-0.5.2-*.tar.gz
    $ cp ripgrep-0.5.2-*/rg $HOME/bin/rg
If you're not a fan of this approach (downloading random binaries and slapping them into your $HOME/bin), then your other choice is to build from source. I don't use Ubuntu, but install Rust[1] and then building ripgrep is easy:

    $ git clone git://github.com/BurntSushi/ripgrep
    $ cd ripgrep
    $ cargo build --release
    $ ./target/release/rg -V
    ripgrep 0.5.2
And yes, it would be great to get ripgrep packaged into Ubuntu.[2] There seems to be an up-to-date PPA here.[3]

[1] - https://www.rust-lang.org/en-US/install.html

[2] - https://github.com/BurntSushi/ripgrep/issues/10

[3] - https://launchpad.net/~x4121/+archive/ubuntu/ripgrep

Re: Skip grep, use awk

#105

Post by gmfawcett »

d33 wrote: post #91
Ultimatt wrote: post #88

Skip awk, use perl.... The alias below sets perl to loop over STDIN splitting each line on more than one whitespace character and populate an array F. The -nE will then Evaluate an expression from the command line looping over the input line-by-line. alias glorp='perl -aF"/\s+/" -nE' So now we have the command `glorp` to play with which has more familiar syntax than awk and all of CPAN available to play with! $ [data…

Hey, it would be awesome to have something like this for Python!

I wrote a little tool called "pyline" about a hundred years ago... I still reach for it often when I'm in a hurry, and don't have time to futz around with awk/sed.

https://code.activestate.com/recipes/437932-pyline-a-grep-li...

Re: Skip grep, use awk

#106

Post by geophile »

d33 wrote: post #91
Ultimatt wrote: post #88

Skip awk, use perl.... The alias below sets perl to loop over STDIN splitting each line on more than one whitespace character and populate an array F. The -nE will then Evaluate an expression from the command line looping over the input line-by-line. alias glorp='perl -aF"/\s+/" -nE' So now we have the command `glorp` to play with which has more familiar syntax than awk and all of CPAN available to play with! $ [data…

Hey, it would be awesome to have something like this for Python!

Check out osh: https://github.com/geophile/osh

Re: Skip grep, use awk

#107

Post by xorcist »

OskarS wrote: post #53
emmelaich wrote: post #40

Earlier quoted context omitted.

As per other users, you can use pkill but it's not entirely portable in meaning between OSs. This software had to run on just about every possible Unix out there. Solaris, HP-UX, AIX, many others ... My main point is that using perl and grep, (multiple thereof!) is nuts when you can do it all in perl. Also crazy was using -ef (all processes) when the process of interest was using a known user. So ps -u would be more…

If it's something you need to do on a semi-regular basis, using a pidfile seems to me to be the best solution. Instead of trying to be clever about pipelines, just write the pid(s) to a file and use that.

Or use a dedicated uid for the daemon. You probably want that anyway.

Re: Skip grep, use awk

#108

Post by xorcist »

desdiv wrote: post #38
stephen82 wrote: post #33

Earlier quoted context omitted.

I tend to use the following command: kill -HUP $(pidof SomeDaemon) and if it insists on running, I use: sudo kill -9 $(pidof SomeDaemon) That's it, really.

I suggest: killall -HUP SomeDaemon

Don't do this! Not only is pkill more competent, it is available on several other operating systems.

There is a killall on Solaris also which is very different but true to its name. You do not want to run it by accident.

Re: Skip grep, use awk

#109

Post by codesnik »

Ultimatt wrote: post #88

Skip awk, use perl.... The alias below sets perl to loop over STDIN splitting each line on more than one whitespace character and populate an array F. The -nE will then Evaluate an expression from the command line looping over the input line-by-line. alias glorp='perl -aF"/\s+/" -nE' So now we have the command `glorp` to play with which has more familiar syntax than awk and all of CPAN available to play with! $ [data…

what's the reason of using -F option? -a defaults " " separator, which already emulates awk behavior. If I'm not wrong, only difference between perl -aF"/\s+/" and perl -a is treating of leading whitespace in lines

Re: Skip grep, use awk

#110

Post by kolodny »

vidarh wrote: post #96
Ultimatt wrote: post #88

Skip awk, use perl.... The alias below sets perl to loop over STDIN splitting each line on more than one whitespace character and populate an array F. The -nE will then Evaluate an expression from the command line looping over the input line-by-line. alias glorp='perl -aF"/\s+/" -nE' So now we have the command `glorp` to play with which has more familiar syntax than awk and all of CPAN available to play with! $ [data…

Here's the equivalent for Ruby: alias glorp='ruby -ane ' $ [data is generated] | glorp ' ~ /Something/ and puts $F[2]' Or: $ echo -e '{"hello":"world"}\n{"hello":"cat"}' | glorp 'puts JSON.load($_)["hello"] ' -rjson (Of course Ruby got the -a autosplit-mode and the -n assumed 'while gets(); ... end' loop from Perl along with $_ and $F, so it's very intentional that they're similar)

Somewhat related nodejs self plug: Use nip https://github.com/kolodny/nip

    $  echo -e 'this\nis\na\nwhatever foo' | nip 'return /whatever/.test(line) && cols[1]' # foo
Post reply on HN