Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for exploring data

datavu.blogspot.com

Post reply on HN
Print view View original

101–110 of 156 posts

Re: Useful Unix commands for exploring data

#101

Post by vram22 »

CraigJPerry wrote: post #3

>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There…

The classic book "The UNIX Programming Environment" by Kernighan and Pike, has a tool in it, called 'overwrite', that does this - letting you safely overwrite a file with the result of a oommand or pipeline, IIRC.

Re: Useful Unix commands for exploring data

#102

Post by smorrow »

mprovost wrote: post #93
bch wrote: post #79

Earlier quoted context omitted.

"Don't pipe a cat" is how I'm used to describing what you're talking about -- it may have been a performance issue in days past, but these days I think it's simply a matter of style. Not that style is not important.

This was drilled into me back in the usenet days. If you see a cat command with a single argument it's almost always replaceable by a shell redirection, or in this case just by passing the filename as an argument to grep. If you're processing lots of data like in the article there's no point in passing it through a separate command and pipe first.

I think people like reading cat thefile | grepsedawk -opts 'prog' from left to right, and that they think the only alternative is grepsedawk -opts 'prog' thefile.

But there's grep <thefile -opts 're'. I like that one best; it reads the same way you'd tend to think it.

Re: Useful Unix commands for exploring data

#104

Post by smorrow »

LiveTheDream wrote: post #61

I use this command very frequently to check how often an event occurs in a log file over time (specifically in 10-minute buckets), assuming the file is formatting like "INFO - [2014-08-27 16:16:29,578] Something something something" cat /path/to/logfile | grep PATTERN | sed 's/.*\(2014-..-..\) \(..\):\(.\).*/\1 \2:\3x/' | uniq -c results in: 273 2014-08-27 14:5x 222 2014-08-27 15:0x 201 2014-08-27 15:1x 171 2014-08-2…

sed -E for awk or grep -E regexes.

Re: Useful Unix commands for exploring data

#105

Post by derekp7 »

epistasis wrote: post #38
zo1 wrote: post #32

"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how yo…

Yeah, I think calling him the "unenlightened" one is pretty off base here. For performing the tasks outlined by his examples, Unix utilities are easier for the user as well as executing faster than writing your own code in a general purpose programming language, unless one puts in the time to tune the implementation. One could rebuild AWK in C and get similar performance, but why not just use some extremely simple AW…

This is one area where I wish the Unix philosophy (reuse of tools) was taken a bit further. Too me, every command should be callable as a C library function. That way you wouldn't have to parse the human readable output through a pipe. Not only that, there needs to be both human-readable, as well as machine-readable output to all commands. For example I would love to be able to call "ps" from another script and easily select specific columns from an xml or json output.

Re: Useful Unix commands for exploring data

#106

Post by vram22 »

WestCoastJustin wrote: post #53

My personal favorite is to use this pattern. You can do some extremely cool counts and group by operations at the command like [1]: grep '01/Jul/1995' NASA_access_log_Jul95 | awk '{print $1}' | sort | uniq -c | sort -h -r | head -n 15 Turns this: 199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245 unicomp6.unicomp.net - - [01/Jul/1995:00:00:06 -0400] "GET /shuttle/countdown/ HTTP/1.…

I think there's something like that in the Kernighan and Pike book I referred to elsewhere in this thread, and also, that code looks similar to this technique:

http://en.wikipedia.org/wiki/Decorate-sort-undecorate

, i.e. Decorate-Sort-Undecorate (DSU), related to the Schwartzian transform.

Re: Useful Unix commands for exploring data

#107

Post by andreasvc »

unhammer wrote: post #96
namplaa wrote: post #86

Earlier quoted context omitted.

Personally I find silversearcher (ag) faster, and reinventing standard commandline tools with a collection of other tools is often slower. One liner shell commands often turn complicated quickly.

ag is often faster when you're using it interactively, replacing "grep -r" (in particular in version controlled dirs). It's also faster in the sense that for interactive use it will often DWYM. But has too many weird quirks that it can replace grep for data munging. E.g. $ ag verb fullform_nn.txt >/tmp/verbs ERR: Too many matches in fullform_nn.txt. Skipping the rest of this file. Man ag says there's a --max-count op…

Did you report it as a bug?

Re: Useful Unix commands for exploring data

#108

Post by zackbloom »

zo1 wrote: post #32

"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how yo…

Just to point out, "big genetic datasets" can be hundreds of gigabytes to terabytes, sizes which are not possible to process easily in any language. A highly optimized C program like uniq might be the fastest option short of a distributed system.

Re: Useful Unix commands for exploring data

#109

Post by dpeck »

Terretta wrote: post #82
dpeck wrote: post #62

Earlier quoted context omitted.

No the case with ruby at all, if you're reading the whole file into memory theres a good chance you're doing it wrong. check out yield and blocks

> if you're reading the whole file into memory theres a good chance you're doing it wrong GP: "working with file lines _ naively _ in Ruby"

ah, my bad, read that as natively and chalked it up to odd wording.

Re: Useful Unix commands for exploring data

#110

Post by vram22 »

jeroenjanssens wrote: post #70
emeraldd wrote: post #4

uniq also doesn't deal well with duplicate records that aren't adjacent. You may need to do a sort before using it. sort | uniq But that can screw with your header lines, so be careful there two.

Try `body`: https://github.com/jeroenjanssens/data-science-at-the-comman... $ echo 'header\ne\nd\na\nb\nc\nb' | body sort | body uniq header a b c d e

Heh, at first I thought you meant a body command like this one, which I've written in the past:

$ cat body_test.txt

     1  This
     2  is
     3  a
     4  file
     5  to
     6  test
     7  the
     8  body
     9  command
    10  which
    11  is
    12  a
    13  complement
    14  to
    15  head
    16  and
    17  tail.
$ cat `which body`

sed -n $1,$2p $3

$ body 5 10 body_test.txt

     5	to
     6	test
     7	the
     8	body
     9	command
    10	which
Post reply on HN