Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for data science

gregreda.com

Post reply on HN
Print view View original

31–40 of 108 posts

Re: Useful Unix commands for data science

#31

Post by foobarbazqux »

daemon13 wrote: post #17
minimax wrote: post #9

AWK is worth learning completely. It hits a real sweet spot in terms of minimizing the number of lines of code needed to write useful programs in the world of quasi-structured (not quite CSV but not completely free form) data. You can learn the whole language and become proficient in an afternoon. I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger, though it seems to be listed for a hilariou…

I recall there was a pointer to an old great AWK tutorial some time ago - smth along the lines 'how to approach awk language....' - anyone kept the link?

This is the first hit for awk tutorial and it's all you need.

http://www.grymoire.com/Unix/Awk.html

Re: Useful Unix commands for data science

#32

Post by makmanalp »

Everyone forgets the brilliant and sometimes crazy BSD ones:

  - Column: Create columns / tables from input data
  - tr: substitute / delete chars
  - join: like a database join, but for text files
  - comm: like diff, but you can use it programmatically to choose if a       line is in one file, or another, or both.
  - paste: put file lines side-by-side
  - rs: reshape arrays
  - jot: generate random or sequence data
  - expand: replace tabs / spaces

Re: Useful Unix commands for data science

#33

Post by D9u »

pmelendez wrote: post #13
gnosis wrote: post #2

I was hoping to see an article about some neat new utilities specifically tailored for doing advanced data analysis. Instead this is a set of basic examples of bog-standard tools that every newbie *nix user should be already familiar with: cat, awk, head, tail, wc, grep, sed, sort, uniq

>"tools that every newbie nix user should be already familiar with" The key word is should ... you might be surprised how many "not newbie" nix users are not aware of those commands or how using them in this fashion. Specially awk.

I forgot all about wc, but thanks to this article I may remember it the next time I need it.

Thanks!

Re: Useful Unix commands for data science

#34

Post by wiredfool »

makmanalp wrote: post #32

Everyone forgets the brilliant and sometimes crazy BSD ones: - Column: Create columns / tables from input data - tr: substitute / delete chars - join: like a database join, but for text files - comm: like diff, but you can use it programmatically to choose if a line is in one file, or another, or both. - paste: put file lines side-by-side - rs: reshape arrays - jot: generate random or sequence data - expand: replace…

Join is really one of those awesome unknown commands.

Re: Useful Unix commands for data science

#35

Post by mjn »

makmanalp wrote: post #32

Everyone forgets the brilliant and sometimes crazy BSD ones: - Column: Create columns / tables from input data - tr: substitute / delete chars - join: like a database join, but for text files - comm: like diff, but you can use it programmatically to choose if a line is in one file, or another, or both. - paste: put file lines side-by-side - rs: reshape arrays - jot: generate random or sequence data - expand: replace…

Looks like 6 of those 8 are in GNU coreutils as well (and therefore can be assumed present on just about any modern Unix). 'rs' and 'jot' are the two missing from most default Linux installs. On Debian you can install them via the packages 'rs' and 'athena-jot'.

Re: Useful Unix commands for data science

#36

Post by [deleted] »

makmanalp wrote: post #32

Everyone forgets the brilliant and sometimes crazy BSD ones: - Column: Create columns / tables from input data - tr: substitute / delete chars - join: like a database join, but for text files - comm: like diff, but you can use it programmatically to choose if a line is in one file, or another, or both. - paste: put file lines side-by-side - rs: reshape arrays - jot: generate random or sequence data - expand: replace…

[deleted]

Re: Useful Unix commands for data science

#37

Post by kingmanaz »

minimax wrote: post #9

AWK is worth learning completely. It hits a real sweet spot in terms of minimizing the number of lines of code needed to write useful programs in the world of quasi-structured (not quite CSV but not completely free form) data. You can learn the whole language and become proficient in an afternoon. I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger, though it seems to be listed for a hilariou…

>I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger

I concur with this recommendation. "The AWK Programming Language", at little over 100 pages, is a classic of programming language instruction. The book jumps right into use cases, it does not waste one's time. This book should be required reading for anyone contemplating writing a handbook on any programming language; my CS bookshelf would be several feet thinner and several times more informative.

Re: Useful Unix commands for data science

#38

Post by jlgaddis »

gnosis wrote: post #2

I was hoping to see an article about some neat new utilities specifically tailored for doing advanced data analysis. Instead this is a set of basic examples of bog-standard tools that every newbie *nix user should be already familiar with: cat, awk, head, tail, wc, grep, sed, sort, uniq

Lots of people are familiar with the "basics" of each of these commands but many of them (awk, sed) are very powerful utilities that can do much, much more than it appears at first glance.

Re: Useful Unix commands for data science

#39

Post by cjg_ »

kingmanaz wrote: post #37
minimax wrote: post #9

AWK is worth learning completely. It hits a real sweet spot in terms of minimizing the number of lines of code needed to write useful programs in the world of quasi-structured (not quite CSV but not completely free form) data. You can learn the whole language and become proficient in an afternoon. I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger, though it seems to be listed for a hilariou…

>I recommend "The AWK Programming Language" by Aho, Kernighan, and Weinberger I concur with this recommendation. "The AWK Programming Language", at little over 100 pages, is a classic of programming language instruction. The book jumps right into use cases, it does not waste one's time. This book should be required reading for anyone contemplating writing a handbook on any programming language; my CS bookshelf would…

Sadly it seems very expensive now, $95 on Amazon...

Re: Useful Unix commands for data science

#40

Post by jlgaddis »

res0nat0r wrote: post #7

If this interests you, you should check out Joyents new Manta service which lets you do this type of thing on your data via their infrastructure. It's really cool. http://www.joyent.com/products/manta

If I needed to do this type of thing on 10 TB of data, it would probably take me longer to get the data to them than it would to just run it on my own hardware.

Apparently there's a need for it, though, or it wouldn't exist.

Post reply on HN