Useful Unix commands for data science • Hacker News Board

Re: Useful Unix commands for data science

#101

Post by FreeFull » Tue, Jul 16, 2013, 2:01 PM UTC

atondwal wrote: post #41

jbert wrote: post #3

I like slicing and dicing with awk, grep and friends too. One thing I find odd that you have to drop to a full language (awk, perl etc) to sum a column of numbers. Am I missing a utility? echo "1\n2\n3\n" | sum # should print 6 with hyphothetical sum command I suppose more generally you could have a 'fold initial op' and: echo "1\n2\n3\n4\n" | fold 0 + # should print 10 echo "1\n2\n3\n4\n" | fold 1 \* # should print…

echo "1\n2\n3\n" | tr '\n' + | bc

Doesn't work. By default, echo doesn't translate \n into a newline, so you have to add the -e flag. Then, bc doesn't like the extra plusses at the end, so you have to either add the -n flag to echo and remove the last \n, or somehow trim the newlines from the end beforehand.

Re: Useful Unix commands for data science

#102

Post by username42 » Tue, Jul 16, 2013, 2:39 PM UTC

minimax wrote: post #9

AWK is worth learning completely. It hits a real sweet spot in terms of minimizing the number of lines of code needed to write useful programs in the world of quasi-structured (not quite CSV but not completely free form) data. You can learn the whole language and become proficient in an afternoon. I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger, though it seems to be listed for a hilariou…

I advise not learning more than basic usage of awk and to spend the time on more versatile languages. You can do very neat tricks with sed and awk, but when the problems become more complex, it is a lot faster to use a smarter language. And if you know well this language, you will discover that it may also be very concise for relatively simpler tasks.

When Perl was created, one of its advertised goal was to avoid all the time lost trying to work around the limitations of awk, sed and shell.

Re: Useful Unix commands for data science

#103

Post by username42 » Tue, Jul 16, 2013, 2:42 PM UTC

EvanKelly wrote: post #69

minimax wrote: post #9

AWK is worth learning completely. It hits a real sweet spot in terms of minimizing the number of lines of code needed to write useful programs in the world of quasi-structured (not quite CSV but not completely free form) data. You can learn the whole language and become proficient in an afternoon. I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger, though it seems to be listed for a hilariou…

Here is Kernighan's personal help file on AWK: http://www.cs.princeton.edu/courses/archive/spr08/cos333/awk... It deals with things he forgets or needs to remind himself of. If you're interested in his other personal tutorials, they are here: http://www.cs.princeton.edu/courses/archive/spr08/cos333/tut...

Kernighan's personal help file is excellent. If you need more, you should switch to a more powerful language.

Re: Useful Unix commands for data science

#104

Post by username42 » Tue, Jul 16, 2013, 2:50 PM UTC

pseut wrote: post #99

dredmorbius wrote: post #97

Earlier quoted context omitted.

Or you can actually use the Linux commands by installing Cygwin. Pretty much my first conscious action when I wake up stranded on a desert Windows system.

Why Cygwin instead of MinGW? (I can't remember why I prefer MinGW, but at some point I had a reason).

Virtualbox ubuntu.

Re: Useful Unix commands for data science

#105

Post by jbert » Tue, Jul 16, 2013, 8:33 PM UTC

FreeFull wrote: post #101

atondwal wrote: post #41

Earlier quoted context omitted.

echo "1\n2\n3\n" | tr '\n' + | bc

Doesn't work. By default, echo doesn't translate \n into a newline, so you have to add the -e flag. Then, bc doesn't like the extra plusses at the end, so you have to either add the -n flag to echo and remove the last \n, or somehow trim the newlines from the end beforehand.

Thanks for the detail. I was worried about the escapes in the echo, but didn't check.

Re: Useful Unix commands for data science

#106

Post by emmelaich » Wed, Jul 17, 2013, 2:14 AM UTC

merlincorey wrote: post #98

mjn wrote: post #35

Earlier quoted context omitted.

Looks like 6 of those 8 are in GNU coreutils as well (and therefore can be assumed present on just about any modern Unix). 'rs' and 'jot' are the two missing from most default Linux installs. On Debian you can install them via the packages 'rs' and 'athena-jot'.

'jot' is pretty sweet, especially for creating ranges for iteration and random numerical data for testing arguments and such. Check out the man page for a few snippets: http://www.unix.com/man-page/FreeBSD/1/jot/ It is the older, more flexible uncle of gnu's 'seq' command: http://administratosphere.wordpress.com/2009/01/23/using-bsd...

And you can't mention jot and rs without lam: http://www.unix.com/man-page/FreeBSD/1/lam/

Re: Useful Unix commands for data science

#107

Post by dredmorbius » Wed, Jul 17, 2013, 4:50 AM UTC

pseut wrote: post #99

dredmorbius wrote: post #97

Earlier quoted context omitted.

Or you can actually use the Linux commands by installing Cygwin. Pretty much my first conscious action when I wake up stranded on a desert Windows system.

Why Cygwin instead of MinGW? (I can't remember why I prefer MinGW, but at some point I had a reason).

I could care less which you choose so long as you're getting a proper Linux toolset.

Powershell is a skill I don't have yet which carries over to ... precisely one declining technical dinosaur (with a penchant for expiring its skillsets).

The Linux toolbox is a set of skills I embarked on learning over a quarter-century ago, most of which goes back another decade or further (the 'k' in 'awk' comes from Brian Kernighan, one of Unix's creators). And while some old utilities are retired and new ones replace them (telnet / rsh for ssh, sccs/rcs for git), much of the core has remained surprisingly stable over time.

The main difference between MinGW and Cygwin appears to be how Windows-native they are considered, which for my own purposes has been an entirely irrelevant distinction, though if you're building applications based off of the tools might matter to you.

http://www.mingw.org/node/21

Re: Useful Unix commands for data science

#108

Post by dap » Wed, Jul 17, 2013, 5:10 PM UTC

jrn wrote: post #96

mcavage wrote: post #56

Earlier quoted context omitted.

Disclaimer: I work at Joyent, on Manta. This entire HN thread is a perfect example of why we built Manta. Lots of engineers/scientists/sysadmins/... already know how to (elegantly) process data using Unix and augmenting with scripts. Manta isn't about always needing to work on a 10TB dataset (you can), but about it being always available, and stored ready to go. I know we can't live without it for running our own sys…

I know manta, has default software packaged, but is it possible to install your own like ghci, or julia? Or is that something that needs to be brought in as an asset. This isn't necessarily a feature request, just trying to figure out how it works. https://apidocs.joyent.com/manta/compute-instance-software.h...

An asset is currently the way to do that.