How to Clean Text Data at the Command Line
ezzeddinabdullah.com
How to Clean Text Data at the Command Line
1–10 of 53 posts
Re: How to Clean Text Data at the Command Line
#2It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocation would do the job without fuss. For instance, TFA uses
awk '{print $1","$2}' | sed '1i count,word'
awk 'BEGIN { print "count,word" } { print $1","$2 }'Re: How to Clean Text Data at the Command Line
#3I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is docker only for the csvlookup tool?
Not trying to be a supermassive asshole but I do feel like I'm losing my mind reading this... In order to use GNU/Linux one must install Linux, setup a container runtime, and then! Use the same utilities available to them in the runtime... from a container?
I feel like if this is the state of a tutorial for someone new to these utilities we're all pretty fucked and should probably take a step back to re-evaluate the mess we've made. I'll go crawl back into my hole now to do just that.
Re: How to Clean Text Data at the Command Line
#4I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…
I was also wondering why it starts with docker. But since this post is targeted for data science or becoming data science folk maybe the idea is to also support windows users?
Re: How to Clean Text Data at the Command Line
#5I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…
I was also wondering why it starts with docker. But since this post is targeted for data science or becoming data science folk maybe the idea is to also support windows users?
Would it not work in WSL though? If you wanted windows support.
Re: How to Clean Text Data at the Command Line
#6I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…
I was also baffled. However, if you follow the link to the "why we use docker" post you'll see the author is using a Mac and had difficulty before while following instructions written for Linux.
Data scientists using Macs for everything and sometimes needing to run Linux in a container to get work done is more common than you might think.
Re: How to Clean Text Data at the Command Line
#7I... am... just like baffled at why Docker is used. Are any of these like not normal unix utilities? What dependencies are you pulling in after GNU/Linux... which is where CONTAINERS come from? I'm even only even putting "GNU" here because it's literally every fucking Linux distro on the planet containing the utilities referenced. Is it common place now-a-days to bloat a docker image with text data or something? Is d…
I was also baffled. However, if you follow the link to the "why we use docker" post you'll see the author is using a Mac and had difficulty before while following instructions written for Linux. Data scientists using Macs for everything and sometimes needing to run Linux in a container to get work done is more common than you might think.
brew install coreutils
Re: How to Clean Text Data at the Command Line
#8It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocation would do the job without fuss. For instance, TFA uses awk '{print $1","$2}' | sed '1i count,word' when you can just add a BEGIN block: awk 'BEGIN { print "count,word" } { print $1","$2 }'
Speaking of AWK, is there a more succinct version of the AWK programming language book that I can use for learning?
https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...
Re: How to Clean Text Data at the Command Line
#9It seems lots of people's knowledge of awk is limited to printing fields, and they'll happily chain awk with a bunch of grep and sed when a single awk invocation would do the job without fuss. For instance, TFA uses awk '{print $1","$2}' | sed '1i count,word' when you can just add a BEGIN block: awk 'BEGIN { print "count,word" } { print $1","$2 }'
Speaking of AWK, is there a more succinct version of the AWK programming language book that I can use for learning? https://ia802309.us.archive.org/25/items/pdfy-MgN0H1joIoDVoI...
You can use the notes of Bruce Barnett for quick lookups:
Re: How to Clean Text Data at the Command Line
#10Earlier quoted context omitted.
I was also wondering why it starts with docker. But since this post is targeted for data science or becoming data science folk maybe the idea is to also support windows users?
Would it not work in WSL though? If you wanted windows support.
Sure. It was just an idea why docker was choosen :)