Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for exploring data

datavu.blogspot.com

Post reply on HN
Print view View original

151–156 of 156 posts

Re: Useful Unix commands for exploring data

#151

Post by faulteh »

vram22 wrote: post #110
jeroenjanssens wrote: post #70

Earlier quoted context omitted.

Try `body`: https://github.com/jeroenjanssens/data-science-at-the-comman... $ echo 'header\ne\nd\na\nb\nc\nb' | body sort | body uniq header a b c d e

Heh, at first I thought you meant a body command like this one, which I've written in the past: $ cat body_test.txt 1 This 2 is 3 a 4 file 5 to 6 test 7 the 8 body 9 command 10 which 11 is 12 a 13 complement 14 to 15 head 16 and 17 tail. $ cat `which body` sed -n $1,$2p $3 $ body 5 10 body_test.txt 5 to 6 test 7 the 8 body 9 command 10 which

Immediately ^'d this post for it's usefulness, but I put this in my .bashrc instead:

function body_alias() {

  sed -n $1,$2p $3
}

alias body=body_alias

I used to have little scripts like body in my own ~/bin or /usr/local/bin but I've been slowly moving those to my .bashrc which I can copy to new systems I log on to.

Re: Useful Unix commands for exploring data

#152

Post by etrain »

collyw wrote: post #141
etrain wrote: post #127

Some more tips from someone who does this every day. 1) Be careful with CSV files and UNIX tools - most big CSV files with text fields have some subset of fields that are text quoted and character-escaped. This means that you might have "," in the middle of a string. Anything (like cut or awk) that depends on comma as a delimiter will not handle this situation well. 2) "cut" has shorter, easier to remember syntax tha…

Actually I would say Perl is more appropriate. I went back to Perl after 4 years for this sort of task, as it has so many features built into the syntax. Plus it can be run as a one liner.

I'm reminded of the old joke, "python is executable pseudocode, while perl is executable line noise."

But seriously, I've got some battle scars from the perl days, and hope not to revisit them. Honestly, there's very little I find I can do with perl and not python, and it's just as easy to express (if not quite as concise) and much simpler to maintain.

But, use the tool that works for you!

Re: Useful Unix commands for exploring data

#153

Post by collyw »

etrain wrote: post #152
collyw wrote: post #141

Earlier quoted context omitted.

Actually I would say Perl is more appropriate. I went back to Perl after 4 years for this sort of task, as it has so many features built into the syntax. Plus it can be run as a one liner.

I'm reminded of the old joke, "python is executable pseudocode, while perl is executable line noise." But seriously, I've got some battle scars from the perl days, and hope not to revisit them. Honestly, there's very little I find I can do with perl and not python, and it's just as easy to express (if not quite as concise) and much simpler to maintain. But, use the tool that works for you!

I use Python and Django most of the time, and its true, you can do pretty much the same thing in each language. But for quick hacky stuff manipulating the filesystem a lot, Perl has many more features built into the language. Things like regex syntax, globing directories, back ticks to execute Unix commands, and the fact you can use it directly from the command line as a one liner. You can do all these (except the last one?) in Python, but Perl is quicker.

Re: Useful Unix commands for exploring data

#154

Post by vram22 »

faulteh wrote: post #151
vram22 wrote: post #110

Earlier quoted context omitted.

Heh, at first I thought you meant a body command like this one, which I've written in the past: $ cat body_test.txt 1 This 2 is 3 a 4 file 5 to 6 test 7 the 8 body 9 command 10 which 11 is 12 a 13 complement 14 to 15 head 16 and 17 tail. $ cat `which body` sed -n $1,$2p $3 $ body 5 10 body_test.txt 5 to 6 test 7 the 8 body 9 command 10 which

Immediately ^'d this post for it's usefulness, but I put this in my .bashrc instead: function body_alias() { sed -n $1,$2p $3 } alias body=body_alias I used to have little scripts like body in my own ~/bin or /usr/local/bin but I've been slowly moving those to my .bashrc which I can copy to new systems I log on to.

Glad you liked it. Your alias technique is good too. Plus it may save a small amount of time since the body script does not have to be loaded from its file (as in my case) - unless your *nix version caches it in memory after the first time.

Re: Useful Unix commands for exploring data

#155

Post by vram22 »

collyw wrote: post #153
etrain wrote: post #152

Earlier quoted context omitted.

I'm reminded of the old joke, "python is executable pseudocode, while perl is executable line noise." But seriously, I've got some battle scars from the perl days, and hope not to revisit them. Honestly, there's very little I find I can do with perl and not python, and it's just as easy to express (if not quite as concise) and much simpler to maintain. But, use the tool that works for you!

I use Python and Django most of the time, and its true, you can do pretty much the same thing in each language. But for quick hacky stuff manipulating the filesystem a lot, Perl has many more features built into the language. Things like regex syntax, globing directories, back ticks to execute Unix commands, and the fact you can use it directly from the command line as a one liner. You can do all these (except the la…

>But for quick hacky stuff manipulating the filesystem a lot, Perl has many more features built into the language. Things like regex syntax, globing directories, back ticks to execute Unix commands

All good points.

>you can use it directly from the command line as a one liner. You can do all these (except the last one?) in Python

You can use Python from the command line too, but Perl has more features for doing that, like the -n and -p flags. Then again, Python has the fileinput module. Here's an example:

http://jugad2.blogspot.in/2013/05/convert-multiple-text-file...

Post reply on HN