Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Clean Text Data at the Command Line

ezzeddinabdullah.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: How to Clean Text Data at the Command Line

#11

Post by asicsp »

mellow-lake-day wrote: post #8
oefrha wrote: post #2

It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocation would do the job without fuss. For instance, TFA uses awk '{print $1","$2}' | sed '1i count,word' when you can just add a BEGIN block: awk 'BEGIN { print "count,word" } { print $1","$2 }'

Speaking of AWK, is there a more succinct version of the AWK programming language book that I can use for learning? https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

I wrote a book on GNU awk one-liners: https://learnbyexample.github.io/learn_gnuawk/

There's plenty of examples and exercises, plus an entire chapter dedicated to regular expressions.

Re: How to Clean Text Data at the Command Line

#12

Post by scottlocklin »

rubyn00bie wrote: post #3

I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…

csvlook/csvkit, apparently; something I never heard of before, which is actually not installed by default in ubuntu. He also included the data itself, which is pretty helpful, but which could have more usefully gone into a git repo somewhere.

Also "kids these days" all reach for docker. Otherwise any sane person would have just used cat instead of csvlook.

Upvoted the kid's article, because command line text cleaning is of TOWERING IMPORTANCE to any DS person worth their salt. But he is definitely an inexperienced kid.

Re: How to Clean Text Data at the Command Line

#13

Post by dredmorbius »

mellow-lake-day wrote: post #8
oefrha wrote: post #2

It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocation would do the job without fuss. For instance, TFA uses awk '{print $1","$2}' | sed '1i count,word' when you can just add a BEGIN block: awk 'BEGIN { print "count,word" } { print $1","$2 }'

Speaking of AWK, is there a more succinct version of the AWK programming language book that I can use for learning? https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...

GNU AWK User's Guide is reasonably succinct, while being fairly comprehensive.

https://www.gnu.org/software/gawk/manual/gawk.html

The manpage alone is quite useful, though lacks some useful details.

The awk FAQ provides general background, including diffeerences between implementations:

http://www.faqs.org/faqs/computer-lang/awk/faq/

Re: How to Clean Text Data at the Command Line

#14

Post by em500 »

inimino wrote: post #6
rubyn00bie wrote: post #3

I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…

I was also baffled. However, if you follow the link to the "why we use docker" post you'll see the author is using a Mac and had difficulty before while following instructions written for Linux. Data scientists using Macs for everything and sometimes needing to run Linux in a container to get work done is more common than you might think.

Professional data scientists should probably do this stuff on company servers, not on their laptops.

Re: How to Clean Text Data at the Command Line

#16

Post by dredmorbius »

oefrha wrote: post #2

It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocation would do the job without fuss. For instance, TFA uses awk '{print $1","$2}' | sed '1i count,word' when you can just add a BEGIN block: awk 'BEGIN { print "count,word" } { print $1","$2 }'

Or do the case-conversion, minimum word-length restriction, and counting in awk itself, here using associative arrays:

  awk  2
        if(length($i)>2) count[$i]++ 
        }
      }; 
    END{ 
      # report
      for(word in count) printf( "%-20s %i\n", word, count[word])
    }
  ' | sort -k2nr -k1
This omits the header (can be trivially added). The external sort can be internalised in gawk using asort(), or by printing to sort via a command:

  cmd="print -2kr -k1"
  for printf(  ) | cmd
  close(cmd)

Re: How to Clean Text Data at the Command Line

#17

Post by leokennis »

isoprophlex wrote: post #7
inimino wrote: post #6

Earlier quoted context omitted.

I was also baffled. However, if you follow the link to the "why we use docker" post you'll see the author is using a Mac and had difficulty before while following instructions written for Linux. Data scientists using Macs for everything and sometimes needing to run Linux in a container to get work done is more common than you might think.

brew install coreutils Lot less complexity than running things in docker.

Was just about to say. What text editing powerhouse from Linux (sed, awk, grep etc.) are not available on Mac? The answer is: none.

Re: How to Clean Text Data at the Command Line

#18

Post by II2II »

larusso wrote: post #4
rubyn00bie wrote: post #3

I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…

I was also wondering why it starts with docker. But since this post is targeted for data science or becoming data science folk maybe the idea is to also support windows users?

If you are interested in platform independence, why even bother with the Unix utilities? You can accomplish the same with Python, have results that are more readable to the non-Unix crowd, and not rely upon sometimes esoteric knowledge of the Unix utilities (and the inconsistencies between vendors).

That isn't to say that the Unix utilities are without merit. I have used them extensively for similar tasks. That being said, it is in environments where everyone used some version of Unix (typically Linux, Solaris, or Mac OS X).

Re: How to Clean Text Data at the Command Line

#19

Post by rgoulter »

rubyn00bie wrote: post #3

I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…

Docker 'can' be used as a way to provide a shell which has a consistent set of programs installed. (As a bonus, using Docker this way let's you avoid 'polluting' the host OS with various packages).

Sure, this case where it's just coreutils + csvlookup, the benefit for the added abstractions isn't so great. Using docker like this makes more sense if there are more dependencies. (Maybe it's just easier to say "use docker" than to explain how to install a dependency on different OSs?).

In general, I think it's fine for people to be able to solve problems without having a full understanding of the mechanisms involved in getting the tool to work. If they need to know it later, they'll learn it then. This is a top-down approach -- But, naturally, there'll be some awkward solutions on the way.

Re: How to Clean Text Data at the Command Line

#20

Post by mdifrgechd »

scottlocklin wrote: post #12
rubyn00bie wrote: post #3

I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…

csvlook/csvkit, apparently; something I never heard of before, which is actually not installed by default in ubuntu. He also included the data itself, which is pretty helpful, but which could have more usefully gone into a git repo somewhere. Also "kids these days" all reach for docker. Otherwise any sane person would have just used cat instead of csvlook. Upvoted the kid's article, because command line text cleaning…

I think the csvlook utility comes from the Command Line Data Science book the author cites. I read some of that book, there was good info in there, but ultimately it was a bit annoying because it's author based a lot of things on utilities he had written himself (like csvlook) which in many cases just did trivial things you could easily have done with shell commands or coreutils.
Post reply on HN