Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Clean Text Data at the Command Line

ezzeddinabdullah.com

Post reply on HN
Print view View original

41–50 of 53 posts

Re: How to Clean Text Data at the Command Line

#41

Post by IanCal »

rubyn00bie wrote: post #22
rgoulter wrote: post #19

Earlier quoted context omitted.

Docker 'can' be used as a way to provide a shell which has a consistent set of programs installed. (As a bonus, using Docker this way let's you avoid 'polluting' the host OS with various packages). Sure, this case where it's just coreutils + csvlookup, the benefit for the added abstractions isn't so great. Using docker like this makes more sense if there are more dependencies. (Maybe it's just easier to say "use dock…

No. I’m sorry but I disagree in this case. These tools are literally installed on every posix machine on the planet. The whole point here is that there are no dependencies this is the “system.” It’s like trying to explain how an engine works by only looking at the turbo, “Oh yeah, gotta have a turbo or your car won’t drive.” ... You mean I gotta have an engine or I can’t even use a turbo? The whole point of my post i…

> If someone is on a Mac, then they are now running a Linux VM, then running containers, then logging into said fucked up environment and then finallly... running... awk?

Err, no, they are not doing this. That's not how it's working for a start (there's no SSH here) but the user experience is literally

Install a standard application, downloaded like any other.

Open terminal.

Run "docker run ..."

That's it.

> It is absolutely 100% only adding complexity and making things more confusing. How many people will not realize these utilities are on every computer they use?

They aren't on every computer, and they are different on different computers. A point the author not only talks about in the linked explanation but describes how that bit them.

Re: How to Clean Text Data at the Command Line

#42

Post by bsagdiyev »

IanCal wrote: post #41
rubyn00bie wrote: post #22

Earlier quoted context omitted.

No. I’m sorry but I disagree in this case. These tools are literally installed on every posix machine on the planet. The whole point here is that there are no dependencies this is the “system.” It’s like trying to explain how an engine works by only looking at the turbo, “Oh yeah, gotta have a turbo or your car won’t drive.” ... You mean I gotta have an engine or I can’t even use a turbo? The whole point of my post i…

> If someone is on a Mac, then they are now running a Linux VM, then running containers, then logging into said fucked up environment and then finallly... running... awk? Err, no, they are not doing this. That's not how it's working for a start (there's no SSH here) but the user experience is literally Install a standard application, downloaded like any other. Open terminal. Run "docker run ..." That's it. > It is ab…

Docker for Mac runs in a VM, so that point still stands. Just because the complexity is hidden away behind a pretty interface doesn’t mean it isn’t there.

Re: How to Clean Text Data at the Command Line

#43

Post by dredmorbius »

hnlmorg wrote: post #36
dredmorbius wrote: post #16

Earlier quoted context omitted.

Or do the case-conversion, minimum word-length restriction, and counting in awk itself, here using associative arrays: awk 2 if(length($i)>2) count[$i]++ } }; END{ # report for(word in count) printf( "%-20s %i\n", word, count[word]) } ' | sort -k2nr -k1 This omits the header (can be trivially added). The external sort can be internalised in gawk using asort(), or by printing to sort via a command: cmd="print -2kr -k1…

If you're going to those lengths then you might as well write it in Go or Python and get the performance boost on larger datasets. However the point often missed when people see awk piped into other CLI tools is that code is intentionally optimised for one time writing rather than maximising awk's usefulness. It's the same with the GPs comment too. In that example the awk code was longer than the awk|sed equivalent.…

For the code here, which I used on an actual dataset (~500 NYT Headlines 1965--1974), the efficiency gaains of a Go / Python rewrite are ... slight.

  real    0m0.176s
  user    0m0.060s
  sys     0m0.060s
On an early-2015 Android tablet running Termux.

I've thrown multimillion row datasets at awk (usually gawk, occasionally mawk, nawk on OSX, and, hell, busybox on occasion) without any practical performance issues. I'm virtually always writing for one-off or project-based analyssys, not live web-scale realtime processing. A second or even ten won't be missed.

I'm also aware that building a pipeline out of grep / cut / sed / tr / sort / uniq / awk is often conceptually nearer at hand. It almost always mirrors how I start exploring some dataset.

But a quick translation to straight awk gives cleaner code, more power, easier conversion to a script, and access to a small library of awk-based utilities I've acumulated.

All with far-more-than-adequate performance.

We've spent far more time discussing this than coding, let alone running, it.

Re: How to Clean Text Data at the Command Line

#44

Post by IanCal »

bsagdiyev wrote: post #42
IanCal wrote: post #41

Earlier quoted context omitted.

> If someone is on a Mac, then they are now running a Linux VM, then running containers, then logging into said fucked up environment and then finallly... running... awk? Err, no, they are not doing this. That's not how it's working for a start (there's no SSH here) but the user experience is literally Install a standard application, downloaded like any other. Open terminal. Run "docker run ..." That's it. > It is ab…

Docker for Mac runs in a VM, so that point still stands. Just because the complexity is hidden away behind a pretty interface doesn’t mean it isn’t there.

> Just because the complexity is hidden away behind a pretty interface doesn’t mean it isn’t there.

In the context of making things hard for a tutorial or complex for a beginner, it not being visible is the key point. Just like the complexities of how your CPU actually works aren't relevant for this tutorial.

Re: How to Clean Text Data at the Command Line

#45

Post by DikshaMA »

"Packers And Movers Bangalore" prompt moving, relocation and shifting services for people and corporation moving to "Bengaluru" and round the India. For Movers Packers Bangalore city full target report on supply of revenue and effective Movers Packers in Bengaluru, contact today 08290173333. We include our network in major cities like Bangalore, Haridwar, Gurgaon, Chandigarh, Delhi NCR, Pune, Hyderabad, Chennai, Noida, Jaipur, Bhubaneswar, Mumbai, Lucknow, Patna, Bhopal, Ahmedabad and Kolkata.@ https://packersmoversbangalore.in/

Re: How to Clean Text Data at the Command Line

#46

Post by hnlmorg »

dredmorbius wrote: post #43
hnlmorg wrote: post #36

Earlier quoted context omitted.

If you're going to those lengths then you might as well write it in Go or Python and get the performance boost on larger datasets. However the point often missed when people see awk piped into other CLI tools is that code is intentionally optimised for one time writing rather than maximising awk's usefulness. It's the same with the GPs comment too. In that example the awk code was longer than the awk|sed equivalent.…

For the code here, which I used on an actual dataset (~500 NYT Headlines 1965--1974), the efficiency gaains of a Go / Python rewrite are ... slight. real 0m0.176s user 0m0.060s sys 0m0.060s On an early-2015 Android tablet running Termux. I've thrown multimillion row datasets at awk (usually gawk, occasionally mawk, nawk on OSX, and, hell, busybox on occasion) without any practical performance issues. I'm virtually al…

> For the code here, which I used on an actual dataset (~500 NYT Headlines 1965--1974), the efficiency gaains of a Go / Python rewrite are ... slight.

500 records isn't a large dataset. Not even close. Large would be orders of millions to billions. And yes, I have had to munge datasets that large on many occasions.

> But a quick translation to straight awk gives cleaner code, more power, easier conversion to a script, and access to a small library of awk-based utilities I've acumulated.

- cleaner: only if you find awk readable. Plenty of people don't. Plenty of people find Go or Python more readable.

- more powerful: again depends. You wouldn't have multithreading builtin like pipelines would. And Python and Go are undoubtedly more powerful than awk. I'm not knocking awk here, just being pragmatic.

- easer conversion to a script: at which point you might as well skip awk entirely and jump straight to Go or Python (or any other programming language)

- and access to a small library of awk-based utilities I've accumulated: that only benefits you. If you're having to sell the benefits of awk to someone then odd are they don't have that small library already to hand ;)

> We've spent far more time discussing this than coding, let alone running, it.

Some of the larger datasets I've had to process have definitely taken longer to munge than my reply here has taken to type :)

Disclaimer: I've honestly not got a problem with awk, I used to use it heavily 20 years ago. But these days its value is diminishing and a lot of the awk evangelists seem to miss the point of why awk isn't well represented in blog posts any more. It's both more verbose than pipelining to coreutils and less powerful than a programming language -- it's that weird middle ground that doesn't provide much value to most people aside those who are already invested into the awk language. Some might see that as a loss but personally I see that as demonstrating the strength of all the other tools we have at our disposal these days.

Re: How to Clean Text Data at the Command Line

#47

Post by dredmorbius »

hnlmorg wrote: post #46
dredmorbius wrote: post #43

Earlier quoted context omitted.

For the code here, which I used on an actual dataset (~500 NYT Headlines 1965--1974), the efficiency gaains of a Go / Python rewrite are ... slight. real 0m0.176s user 0m0.060s sys 0m0.060s On an early-2015 Android tablet running Termux. I've thrown multimillion row datasets at awk (usually gawk, occasionally mawk, nawk on OSX, and, hell, busybox on occasion) without any practical performance issues. I'm virtually al…

> For the code here, which I used on an actual dataset (~500 NYT Headlines 1965--1974), the efficiency gaains of a Go / Python rewrite are ... slight. 500 records isn't a large dataset. Not even close. Large would be orders of millions to billions. And yes, I have had to munge datasets that large on many occasions. > But a quick translation to straight awk gives cleaner code, more power, easier conversion to a script…

I don't have your scale problems. You might care to not impose them where they don't exist.

Re: How to Clean Text Data at the Command Line

#49

Post by ethink »

tinkertamper wrote: post #23

I’ll probably catch a lot of flack but a lot of criticism in these comments for the author who finished something. It’s easy to sit in the cheap seats and cast stones, but this guy made something that works, and you can always make your own if you know better without tearing down the work of others.

Thanks for replying I really appreciate many comments and I respect all opinions except insults. I'm still learning and will always do and I consider my writing as a way to learn more about the field.

Re: How to Clean Text Data at the Command Line

#50

Post by hnlmorg »

dredmorbius wrote: post #47
hnlmorg wrote: post #46

Earlier quoted context omitted.

> For the code here, which I used on an actual dataset (~500 NYT Headlines 1965--1974), the efficiency gaains of a Go / Python rewrite are ... slight. 500 records isn't a large dataset. Not even close. Large would be orders of millions to billions. And yes, I have had to munge datasets that large on many occasions. > But a quick translation to straight awk gives cleaner code, more power, easier conversion to a script…

I don't have your scale problems. You might care to not impose them where they don't exist.

Don't be so ridiculous. You're the one imposing arbitrary problems by saying "everything should be written in awk" then writing an example that was an order of magnitude longer in both character count and execution time than the examples that you were suggesting was wrong.

All I'm doing is citing a few reasons why someone might prefer a terser pipeline but I hadn't realised this wasn't supposed to be an objective conversation and since I have no interest in engaging in pointless language fanboyism I'm just going to leave you to it.

Post reply on HN