Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Useful Unix commands for exploring data

datavu.blogspot.com

Post reply on HN
Print view View original

51–60 of 156 posts

Re: Useful Unix commands for exploring data

#51

Post by icebraining »

aks_c wrote: post #10
CraigJPerry wrote: post #3

>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There…

Thank you for inputs, how about this? uniq -u movies.csv > temp.csv temp.csv > movie.csv rm temp.csv

  $ temp.csv > movie.csv
  temp.csv: command not found

Re: Useful Unix commands for exploring data

#52

Post by mbreese »

lutusp wrote: post #36

Quote: "While dealing with big genetic data sets ..." What a great start. Unless he's a biologist, the author means generic , not genetic . The author goes on to show that he can use command-line utilities to accomplish what database clients do much more easily.

I think that he actually is a biologist. He refers to movies as a parallel universe. In which case, these tools are probably not all that helpful. Biological data is usually in the scale of either "Excel can handle it" (shudder) or "ginormous".

In the later case, none of these would be all that useful, and CSV is not the standard format for most of the biological data that I see.

Databases are less helpful than you'd imagine for this type of data as the schemas are not well defined. I am curious to know how JSON records would work for these data, because I could see something like that working for processing biological data files.

Re: Useful Unix commands for exploring data

#53

Post by WestCoastJustin »

My personal favorite is to use this pattern. You can do some extremely cool counts and group by operations at the command like [1]:

  grep '01/Jul/1995' NASA_access_log_Jul95 | 
    awk '{print $1}' | 
    sort | 
    uniq -c | 
    sort -h -r | 
    head -n 15
Turns this:

  199.72.81.55 - - [01/Jul/1995:00:00:01 -0400] "GET /history/apollo/ HTTP/1.0" 200 6245
  unicomp6.unicomp.net - - [01/Jul/1995:00:00:06 -0400] "GET /shuttle/countdown/ HTTP/1.0" 200 3985
  199.120.110.21 - - [01/Jul/1995:00:00:09 -0400] "GET /shuttle/missions/sts-73/mission-sts-73.html HTTP/1.0" 200 4085
  burger.letters.com - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/countdown/liftoff.html HTTP/1.0" 304 0
  199.120.110.21 - - [01/Jul/1995:00:00:11 -0400] "GET /shuttle/missions/sts-73/sts-73-patch-small.gif HTTP/1.0" 200 4179
Into this:

    623 piweba3y.prodigy.com
    547 piweba4y.prodigy.com
    536 alyssa.prodigy.com
    463 disarray.demon.co.uk
    456 piweba1y.prodigy.com
    417 www-b6.proxy.aol.com
    350 burger.letters.com
    300 poppy.hensa.ac.uk
    279 www-b5.proxy.aol.com
[1] https://sysadmincasts.com/episodes/28-cli-monday-cat-grep-aw...

Re: Useful Unix commands for exploring data

#54

Post by gav »

smorrow wrote: post #44
gav wrote: post #30

Earlier quoted context omitted.

> But that can screw with your header lines, so be careful there two. F=filename; (head -n 1 $F ; tail -n +2 $F | sort -u) | sponge $F To get counts of duplicates, you can use: sort filename | uniq -c | awk '$1 != 1'

If you were piping into that bracketed expression (instead of using a real file), you'd need "line", "9 read", "sh -c 'read ln; echo $ln'", or "bash -c 'read; echo $REPLY'" in place of the head since head, sed, or anything else, might use buffered I/O and bite off more than it chews. (and then a plain cat in place of the tail) "line" will compile anywhere but I only know it to be generally available on Linux. I think…

In the spirit of more options, `pee` comes with moreutils and does the trick:

    cat filename | pee 'head -n 1' 'tail -n +2 | sort -u'

Re: Useful Unix commands for exploring data

#56

Post by icebraining »

jason_slack wrote: post #43

The author states: uniq -u movies.csv > temp.csv mv temp.csv movie.csv **Important thing to note here is uniq wont work if duplicate records are not adjacent. [Addition based on HN inputs] Would the fix here be to sort the lines first using the `sort` command first? Then `uniq`?

Yes, but not first, rather instead. "sort -u" both sorts and hides duplicates.

Re: Useful Unix commands for exploring data

#57

Post by coderholic »

jason_slack wrote: post #43

The author states: uniq -u movies.csv > temp.csv mv temp.csv movie.csv **Important thing to note here is uniq wont work if duplicate records are not adjacent. [Addition based on HN inputs] Would the fix here be to sort the lines first using the `sort` command first? Then `uniq`?

Yes, and in fact you can just use sort's -u (unqiue) argument, and avoid uniq all together.

Re: Useful Unix commands for exploring data

#58

Post by kafkaesk »

lutusp wrote: post #36

Quote: "While dealing with big genetic data sets ..." What a great start. Unless he's a biologist, the author means generic , not genetic . The author goes on to show that he can use command-line utilities to accomplish what database clients do much more easily.

A few blog posts earlier, the author writes about "Network Analysis application in Genetic Studies ", so I am confident to say this isn't a typo.

And for a quick-and-dirty custom analysis of big data sets, the unix tools might be a lot more convenient than databases.

Re: Useful Unix commands for exploring data

#59

Post by snowwrestler »

lutusp wrote: post #36

Quote: "While dealing with big genetic data sets ..." What a great start. Unless he's a biologist, the author means generic , not genetic . The author goes on to show that he can use command-line utilities to accomplish what database clients do much more easily.

He is a biologist.
Post reply on HN