Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Clean Text Data at the Command Line

ezzeddinabdullah.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: How to Clean Text Data at the Command Line

#31

Post by dredmorbius »

derefr wrote: post #29
dredmorbius wrote: post #16

Earlier quoted context omitted.

Or do the case-conversion, minimum word-length restriction, and counting in awk itself, here using associative arrays: awk 2 if(length($i)>2) count[$i]++ } }; END{ # report for(word in count) printf( "%-20s %i\n", word, count[word]) } ' | sort -k2nr -k1 This omits the header (can be trivially added). The external sort can be internalised in gawk using asort(), or by printing to sort via a command: cmd="print -2kr -k1…

When you have such a large awk script, you may as well also demonstrate that Awk is a “real scripting language” by putting it into its own script with a `#!/usr/bin/awk -f` shebang line. Much easier to edit than trying to get your shell to do it. (Unless you’re using a fancy editing shell like Emacs’ shell mode; but at that point, why would you need Awk? You probably know far better how to use your own shell-runtime’…

Sure. Though such scripts often begin as shell one-liners.

This one did. Never left it, as it happens.

C-x C-e FTFW.

(I expanded, indented, and commented the example for HN.)

Re: How to Clean Text Data at the Command Line

#32

Post by leonim »

jhoechtl wrote: post #25
oefrha wrote: post #2

It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocation would do the job without fuss. For instance, TFA uses awk '{print $1","$2}' | sed '1i count,word' when you can just add a BEGIN block: awk 'BEGIN { print "count,word" } { print $1","$2 }'

I beg to differ. I did a lot of csv wrangling on Unix. Csv is a beast. My tools of choice ultimately was miller, an absolutely underrated tool: https://github.com/johnkerl/miller

Totally agree. I think mlr is a wonderful CLI tool. It is very robust, and can handle/convert multiple tabular formats including csv, tsv, json, fixed-format, etc. It has a pretty decent text output formatting, with the --opprint flag.

I use to be very comfortable using awk/sed/perl/sort/uniq/tr/tail/head from the CLI for the sort of data cleaning this article is talking about. However, over the past year I've found I use VisiData https://github.com/saulpw/visidata for interactive work.

If I need to clean up the data first, I'll use mlr or jq as input to Visidata. If my data is too dirty for mlr, then I'll use Unix toolbox tools mentioned as input to mlr, jq or VisiData.

VisiData provides some ability to script, but when possible I prefer to have the shell do the scripting with all the tools mentioned as input to Visidata.

Re: How to Clean Text Data at the Command Line

#33

Post by leokennis »

howlin wrote: post #24
leokennis wrote: post #17

Earlier quoted context omitted.

Was just about to say. What text editing powerhouse from Linux (sed, awk, grep etc.) are not available on Mac? The answer is: none.

There are some rather archaic behaviors of the versions of these tools that come by default on MasOSX. For instance, on a mac the 'sed' command doesn't understand the difference between space and tab https://stackoverflow.com/questions/40311339/using-sed-on-ma...

Oh, very likely true. But nothing stands in your way to just replace them with the GNU versions. Still way easier than running a Docker container...

Re: How to Clean Text Data at the Command Line

#34

Post by IanCal »

rubyn00bie wrote: post #3

I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…

This is a very strong response to what seems like an obvious point clearly laid out at the start.

This is a tutorial and set of examples, and it comes with a fully working environment with example data exactly setup to work. No difference between gsort and sort, no version issues or whether someone is using brew or macports or anything else. No python environment problems or pip Vs pip3 or whatever other random crap you need to deal with when trying to do something simple with lots of varied setups people have.

You are entirely free to not use docker.

> In order to use GNU/Linux one must

Nope. You can. You don't have to.

Re: How to Clean Text Data at the Command Line

#35

Post by IanCal »

isoprophlex wrote: post #7
inimino wrote: post #6

Earlier quoted context omitted.

I was also baffled. However, if you follow the link to the "why we use docker" post you'll see the author is using a Mac and had difficulty before while following instructions written for Linux. Data scientists using Macs for everything and sometimes needing to run Linux in a container to get work done is more common than you might think.

brew install coreutils Lot less complexity than running things in docker.

That won't replace the utils though, will it? You get ones with a g prefix as far as I remember.

Re: How to Clean Text Data at the Command Line

#36

Post by hnlmorg »

dredmorbius wrote: post #16
oefrha wrote: post #2

It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocation would do the job without fuss. For instance, TFA uses awk '{print $1","$2}' | sed '1i count,word' when you can just add a BEGIN block: awk 'BEGIN { print "count,word" } { print $1","$2 }'

Or do the case-conversion, minimum word-length restriction, and counting in awk itself, here using associative arrays: awk 2 if(length($i)>2) count[$i]++ } }; END{ # report for(word in count) printf( "%-20s %i\n", word, count[word]) } ' | sort -k2nr -k1 This omits the header (can be trivially added). The external sort can be internalised in gawk using asort(), or by printing to sort via a command: cmd="print -2kr -k1…

If you're going to those lengths then you might as well write it in Go or Python and get the performance boost on larger datasets.

However the point often missed when people see awk piped into other CLI tools is that code is intentionally optimised for one time writing rather than maximising awk's usefulness.

It's the same with the GPs comment too. In that example the awk code was longer than the awk|sed equivalent.

Don't get me wrong, I am a big fan of awk myself. But sometimes people get so hung up on better usage of awk that they lose sight of the point behind the command line example.

Re: How to Clean Text Data at the Command Line

#37

Post by rubyn00bie »

IanCal wrote: post #34
rubyn00bie wrote: post #3

I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…

This is a very strong response to what seems like an obvious point clearly laid out at the start. This is a tutorial and set of examples, and it comes with a fully working environment with example data exactly setup to work. No difference between gsort and sort, no version issues or whether someone is using brew or macports or anything else. No python environment problems or pip Vs pip3 or whatever other random crap…

> This is a very strong response to what seems like an obvious point clearly laid out at the start.

The point is invalid and clearly untrue.

> This is a tutorial and set of examples, and it comes with a fully working environment with example data exactly setup to work. No difference between gsort and sort, no version issues or whether someone is using brew or macports or anything else. No python environment problems or pip Vs pip3 or whatever other random crap you need to deal with when trying to do something simple with lots of varied setups people have.

I really don't see how having to get Docker going is alleviating any of the above headaches but I'm far removed from these tools being "new to me." Setting up virtualization to run a process on my local machine which I can run without virtualization or containerization the very same way seems crazy to me. It seems like an awful amount of complexity to push onto beginners for what is likely to be zero benefit and gives them only what they had at the start...

> You are entirely free to not use docker. >> In order to use GNU/Linux one must > Nope. You can. You don't have to.

Yeah that's my point exactly... you don't have to at all in fact, Docker offers most literally nothing but opaque complexity. I was saying you MUST have Linux if you want containers because containers are a Linux feature and DNE on MacOS but all them sweet utilities are already there save the weird csv tool. I don't think the complexity of Docker is warranted for simply handling a CSV tool and I find it to be a disservice to anyone who is learning.

/shrug

Re: How to Clean Text Data at the Command Line

#38

Post by IanCal »

rubyn00bie wrote: post #37
IanCal wrote: post #34

Earlier quoted context omitted.

This is a very strong response to what seems like an obvious point clearly laid out at the start. This is a tutorial and set of examples, and it comes with a fully working environment with example data exactly setup to work. No difference between gsort and sort, no version issues or whether someone is using brew or macports or anything else. No python environment problems or pip Vs pip3 or whatever other random crap…

> This is a very strong response to what seems like an obvious point clearly laid out at the start. The point is invalid and clearly untrue. > This is a tutorial and set of examples, and it comes with a fully working environment with example data exactly setup to work. No difference between gsort and sort, no version issues or whether someone is using brew or macports or anything else. No python environment problems…

What point is invalid? That it's a prepackaged setup for all users to avoid having these kinds of problems? It's pointed out right at the beginning and then a longer description is linked to about how the writer hit exactly this kind of problem because mac has different basic utilities and their sed worked differently.

> I really don't see how having to get Docker going is alleviating any of the above headaches

Because it is a single thing to get working that then leaves all users of your tutorial on all major platforms dealing with the exact same environment. Have you ever tried to work with getting end users up to speed only to discover that one is using python2.7, another has anaconda but pip is pointing to the wrong place, pip3 is required for someone else and one user has mac so needs to install brew then replace every standard call to sort with gsort. Then dateutils appears as some conflicted dependency...

> Setting up virtualization to run a process on my local machine which I can run without virtualization or containerization the very same way seems crazy to me

Then don't, you don't have to. Go and setup an identical environment. For those that have docker this becomes a single command to run, and those that don't can get setup quickly. If they're heading into data science they're likely to benefit from having at least a basic understanding of docker.

> It seems like an awful amount of complexity to push onto beginners for what is likely to be zero benefit and gives them only what they had at the start...

It doesn't give zero benefit, it removes a whole bunch of differences in setup that are entirely irrelevant to the tutorial.

> Docker offers most literally nothing but opaque complexity.

It offers a standard environment that puts all users in the same place.

> but all them sweet utilities are already there save the weird csv too

No they are not, not on a mac and not on windows.

> I don't think the complexity of Docker is warranted for simply handling a CSV tool

What complexity? 'docker run --rm -it X bash' is literally all I need to do.

Re: How to Clean Text Data at the Command Line

#39

Post by IanCal »

leokennis wrote: post #33
howlin wrote: post #24

Earlier quoted context omitted.

There are some rather archaic behaviors of the versions of these tools that come by default on MasOSX. For instance, on a mac the 'sed' command doesn't understand the difference between space and tab https://stackoverflow.com/questions/40311339/using-sed-on-ma...

Oh, very likely true. But nothing stands in your way to just replace them with the GNU versions. Still way easier than running a Docker container...

You do however now need to tell all the users of your tutorial to go and install brew, then install coreutils and then to replace every use of "sort" with "gsort", etc.

Running a docker container, regardless of what it does under the hood, is extremely easy on a mac. You install a program then do "docker run ...".

Re: How to Clean Text Data at the Command Line

#40

Post by IanCal »

II2II wrote: post #18
larusso wrote: post #4

Earlier quoted context omitted.

I was also wondering why it starts with docker. But since this post is targeted for data science or becoming data science folk maybe the idea is to also support windows users?

If you are interested in platform independence, why even bother with the Unix utilities? You can accomplish the same with Python, have results that are more readable to the non-Unix crowd, and not rely upon sometimes esoteric knowledge of the Unix utilities (and the inconsistencies between vendors). That isn't to say that the Unix utilities are without merit. I have used them extensively for similar tasks. That being…

> If you are interested in platform independence, why even bother with the Unix utilities? You can accomplish the same with Python,

Because then you're also looking at getting people using the same version of python and your dependencies - and it's likely slower.

> That being said, it is in environments where everyone used some version of Unix (typically Linux, Solaris, or Mac OS X).

If only there was a technology aimed at sharing an exact same version for easy use ;)

Post reply on HN