An introduction to data processing on the Linux command line • Hacker News Board

Re: An introduction to data processing on the Linux command line

#61

Post by gnufx » Sat, Nov 23, 2019, 10:26 PM UTC

clarry wrote: post #37

rodrigo975 wrote: post #12

Why people use Linux in place of *nix ? Even worst, most of the tools (cat, grep, awk) are Unix commands, redeveloped by the GNU project in most of the GNULinux distros.

> Why people use Linux in place of nix ? I find it more irritating when people try to score greybeard points by saying *nix (or Unix) when it's obvious that they're talking about a Linux-only mechanism and quite possibly haven't ever used Unix (or a direct derivative).

Linux is a kernel, which is irrelevant; I've run the GNU tools on many different systems, including MS-DOS, over the years. POSIX now defines UNIX anyway.

Re: An introduction to data processing on the Linux command line

#62

Post by FZ1 » Sat, Nov 23, 2019, 10:58 PM UTC

netmonk wrote: post #30

Ugly UUOC (Useless Use Of Cat). Damn peoples, please i appreciate your will to share, but share good contents and stop spreading bad shell patterns....

UUOC is a pedantic ford-chevy argument.

Re: An introduction to data processing on the Linux command line

#63

Post by School-Cotton » Sat, Nov 23, 2019, 11:09 PM UTC

clarry wrote: post #59

School-Cotton wrote: post #46

Earlier quoted context omitted.

Huh? What is specific to Linux in this post? Also, the most popular Unix-like OS (far more than Linux) is macOS, basically the least “leet greybeard affectation” thing I can imagine. Your irritation is way off base.

> Huh? What is specific to Linux in this post? Perhaps nothing? I was responding to the complaint in general terms. > Your irritation is way off base. Please allow me to feel irritated when people refer to obvious Linux things as something that's supposedly got something to do with Unix. It happens often enough.

Fair enough; I suppose it’d be annoying for people to talk about “the Unix concept of cgroup namespaces” or something like that.

I had thought that you were directly responding to the original poster.

Re: An introduction to data processing on the Linux command line

#64

Post by netmonk » Sat, Nov 23, 2019, 11:13 PM UTC

robertelder wrote: post #32

Hikikomori wrote: post #31

Earlier quoted context omitted.

Yes, it made the whole article useless.

I assume you're just joking around, but to you and the parent comment, I'd be happy to hear any good arguments for avoiding 'useless' cat. Note that I did mentioned 'useless cat' in the article, and there is already a comment thread in this article that contains my opinions on it.

So why should we repeat ?

Re: An introduction to data processing on the Linux command line

#65

Post by mark_l_watson » Sat, Nov 23, 2019, 11:42 PM UTC

dvrx wrote: post #55

mark_l_watson wrote: post #47

Not really where the author is heading, but I like to configure a backend for mathplot lib to render graphics in a terminal so when I am SSHed to a remote system I can get inlined plots.

Better solution: sixel-gnuplot Shameless plug: https://github.com/csdvrx/sixel-gnuplot

Thanks, I will try that.

Re: An introduction to data processing on the Linux command line

#66

Post by criddell » Sat, Nov 23, 2019, 11:51 PM UTC

robertelder wrote: post #38

criddell wrote: post #34

Earlier quoted context omitted.

> Build a pipeline like I've discussed in the article, then pipe it into Python or R. Why not just do it all in Python or R? That way you also get something that will probably work on non-unix platforms.

Over the years I've found that I usually fall into a pattern of starting with low-fidelity automation in languages like shell and slowly re-writing it over time into more higher-level languages, usually python first, then Java. This way, unimportant tasks can be automated in less than 5 min with one of these shell commands. If it breaks or has errors, no big deal. Python works well for figuring out the structure of t…

I go to Python first because it's nice to be able to single-step through the script with a debugger and monitor exactly what's happening. I also know Python a lot better than shell script so it saves me a lot of time as well.

Re: An introduction to data processing on the Linux command line

#67

Post by mnaydin » Sun, Nov 24, 2019, 1:05 AM UTC

ibern wrote: post #33

Here are some ways you could simplify some of the tasks in the article, saving on typing: cat data.csv | sed 's/"//g' can be simplified by doing this instead: cat data.csv | tr d '"' This awk command: cat sales.csv | awk -F',' '{print $1}' | sort | uniq Can be replaced with a simpler (IMO) cut instead: cat sales.csv | cut -d , -f 1 | sort | uniq When using head or tail like this: head -n 3 You don't need the -n: head…

The awk command

  cat sales.csv | awk -F',' '{print $1}' | sort | uniq

can even be further simplified to

  cut -d, -f1 sales.csv | sort -u

Re: An introduction to data processing on the Linux command line

#68

Post by mnaydin » Sun, Nov 24, 2019, 1:29 AM UTC

I wouldn't use awk for simple things such as

  cat sales.csv | awk -F',' '{print $1}'

but I'd prefer

  cut -d, -f1 sales.csv

Re: An introduction to data processing on the Linux command line

#69

Post by dima55 » Sun, Nov 24, 2019, 1:42 AM UTC

mark_l_watson wrote: post #47

Not really where the author is heading, but I like to configure a backend for mathplot lib to render graphics in a terminal so when I am SSHed to a remote system I can get inlined plots.

https://github.com/dkogan/gnuplotlib

Re: An introduction to data processing on the Linux command line

#70

Post by dvrx » Sun, Nov 24, 2019, 3:43 AM UTC

mark_l_watson wrote: post #65

dvrx wrote: post #55

Earlier quoted context omitted.

Better solution: sixel-gnuplot Shameless plug: https://github.com/csdvrx/sixel-gnuplot

Thanks, I will try that.

If you like it, share your terminal configuration!

mlterm works.

mintty had a regression, 3.1.0 may have fixed that