Ugly UUOC (Useless Use Of Cat). Damn peoples, please i appreciate your will to share, but share good contents and stop spreading bad shell patterns....
Yes, it made the whole article useless.
An introduction to data processing on the Linux command line
31–40 of 73 posts
Re: An introduction to data processing on the Linux command line
#32Ugly UUOC (Useless Use Of Cat). Damn peoples, please i appreciate your will to share, but share good contents and stop spreading bad shell patterns....
Yes, it made the whole article useless.
I assume you're just joking around, but to you and the parent comment, I'd be happy to hear any good arguments for avoiding 'useless' cat. Note that I did mentioned 'useless cat' in the article, and there is already a comment thread in this article that contains my opinions on it.
Re: An introduction to data processing on the Linux command line
#33Here are some ways you could simplify some of the tasks in the article, saving on typing:
cat data.csv | sed 's/"//g'
cat data.csv | tr d '"'
cat sales.csv | awk -F',' '{print $1}' | sort | uniq
cat sales.csv | cut -d , -f 1 | sort | uniq
head -n 3
head -3
Re: An introduction to data processing on the Linux command line
#34Can somebody explain the advantage of doing it on the command line vs in Python or R? What would a practical use case look like?
The most significant use case for all things command-line IMHO is automation . Also, I would change that from "command line vs in Python or R" to "command line and Python or R". Build a pipeline like I've discussed in the article, then pipe it into Python or R.
> Build a pipeline like I've discussed in the article, then pipe it into Python or R.
Why not just do it all in Python or R? That way you also get something that will probably work on non-unix platforms.
Re: An introduction to data processing on the Linux command line
#35Can somebody explain the advantage of doing it on the command line vs in Python or R? What would a practical use case look like?
The advantage is that it's faster to prototype/write on the command line and usually ends up being less verbose (although potentially harder to read). It's easy to see what you're data is doing as you work with it and incrementally add pipes to new commands.
I like to use command line tools for for one-off tasks that I'm unlikely to repeat. If there's a task I know I'll need to repeat or is too cumbersome to do in a couple of lines, I'll reach for Python.
Re: An introduction to data processing on the Linux command line
#36Why people use Linux in place of *nix ? Even worst, most of the tools (cat, grep, awk) are Unix commands, redeveloped by the GNU project in most of the GNULinux distros.
Speaking of GNU there’s Datamash https://www.gnu.org/software/datamash/ if you like doing “data science” in the shell
Re: An introduction to data processing on the Linux command line
#37Why people use Linux in place of *nix ? Even worst, most of the tools (cat, grep, awk) are Unix commands, redeveloped by the GNU project in most of the GNULinux distros.
> Why people use Linux in place of nix ?
I find it more irritating when people try to score greybeard points by saying *nix (or Unix) when it's obvious that they're talking about a Linux-only mechanism and quite possibly haven't ever used Unix (or a direct derivative).
Re: An introduction to data processing on the Linux command line
#38Earlier quoted context omitted.
The most significant use case for all things command-line IMHO is automation . Also, I would change that from "command line vs in Python or R" to "command line and Python or R". Build a pipeline like I've discussed in the article, then pipe it into Python or R.
> Build a pipeline like I've discussed in the article, then pipe it into Python or R. Why not just do it all in Python or R? That way you also get something that will probably work on non-unix platforms.
Over the years I've found that I usually fall into a pattern of starting with low-fidelity automation in languages like shell and slowly re-writing it over time into more higher-level languages, usually python first, then Java. This way, unimportant tasks can be automated in less than 5 min with one of these shell commands. If it breaks or has errors, no big deal. Python works well for figuring out the structure of the solution as an actual program, and then finally a language with static type checking when it really needs to run without errors.
Re: An introduction to data processing on the Linux command line
#39Regarding more than one mentions of UUOC in this thread:
- The original award started in 1995. Even though pentium was already out, I think it is safe to say that was the era of 486 PCs. In 2019, for day-to-day shell work (meaning no GBs of file-processing or anything like that), isn't invoking UUOC and pointing out inefficiencies an example of premature optimization [1]?
- Isn't readability a matter of subjectivity, and that for some folks 'cat file' is more readable than '[1] http://wiki.c2.com/?PrematureOptimization
[2] https://chat.stackoverflow.com/rooms/182573/discussion-on-an...
Re: An introduction to data processing on the Linux command line
#40A plug of my tools:
To visualize data coming in from a pipe, can pipe it to
https://github.com/dkogan/feedgnuplot
Very useful in conjunction with other tools to provide filtering and manipulation. For instance (the first one is mine):
https://github.com/dkogan/vnlog
https://www.gnu.org/software/datamash/
https://csvkit.readthedocs.io/
https://github.com/johnkerl/miller
https://github.com/eBay/tsv-utils-dlang
https://github.com/BatchLabs/charlatan
https://github.com/dinedal/textql
https://github.com/BurntSushi/xsv
https://github.com/dbohdan/sqawk