An introduction to data processing on the Linux command line • Hacker News Board

Re: An introduction to data processing on the Linux command line

#51

Post by nurettin » Sat, Nov 23, 2019, 7:48 PM UTC

robertelder wrote: post #10

Hi, (I wrote the article). A few people commented noting that I included "Data Science" in the title, but the content doesn't include any statistics or machine learning which is closer to the core definition of 'data science'. I still think the title is appropriate since any kind of low-fidelity data science task you do on some had-hoc data (log files, heaps of text, web pages) is going to start with setting up a pro…

yes, this article describes - what we used to call in the early 2000s - using linux.

Re: An introduction to data processing on the Linux command line

#52

Post by dang » Sat, Nov 23, 2019, 8:09 PM UTC

robertelder wrote: post #10

Hi, (I wrote the article). A few people commented noting that I included "Data Science" in the title, but the content doesn't include any statistics or machine learning which is closer to the core definition of 'data science'. I still think the title is appropriate since any kind of low-fidelity data science task you do on some had-hoc data (log files, heaps of text, web pages) is going to start with setting up a pro…

Now that you mention it, 'data processing' seems more neutral and accurate, so I've put that in the title above.

Re: An introduction to data processing on the Linux command line

#53

Post by heliodor » Sat, Nov 23, 2019, 8:50 PM UTC

jpxw wrote: post #25

robertelder wrote: post #24

Earlier quoted context omitted.

hah, I knew someone would point that out (which is why I talked about it in the article). I actually prefer useless cat because when you're prototyping a pipeline it's very awkward to use non-useless cat. You'll probably start off with something like this to observe the content of the file: cat something.txt Using this doesn't work in bash: Then, continuing with useless cat to build on it you do cat something.txt | g…

If you’re working with a lot of data you probably want to pipe it into head anyway, initially, so Can be the starting command. When you no longer need the head there, just get rid of “head |”. Although I agree that the pointing out of “useless cat” is usually not particularly useful or constructive.

Rather than using head and worrying about the size of the file, it is easier to simply use "useless" cat then ctrl-c the stream of data that comes out.

Re: An introduction to data processing on the Linux command line

#54

Post by bitminer » Sat, Nov 23, 2019, 9:01 PM UTC

robertelder wrote: post #10

Hi, (I wrote the article). A few people commented noting that I included "Data Science" in the title, but the content doesn't include any statistics or machine learning which is closer to the core definition of 'data science'. I still think the title is appropriate since any kind of low-fidelity data science task you do on some had-hoc data (log files, heaps of text, web pages) is going to start with setting up a pro…

Much can be done just with awk.

My pet peeve is the "grep | awk" idiom. No, just use awk.

Awk does map/reduce, relational joins, associative memory, table lookup, and so on. Just use awk arrays, begin block, and end block.

Re: An introduction to data processing on the Linux command line

#55

Post by dvrx » Sat, Nov 23, 2019, 9:18 PM UTC

mark_l_watson wrote: post #47

Not really where the author is heading, but I like to configure a backend for mathplot lib to render graphics in a terminal so when I am SSHed to a remote system I can get inlined plots.

Better solution: sixel-gnuplot

Shameless plug: https://github.com/csdvrx/sixel-gnuplot

Re: An introduction to data processing on the Linux command line

#56

Post by dvrx » Sat, Nov 23, 2019, 9:30 PM UTC

ibern wrote: post #33

Here are some ways you could simplify some of the tasks in the article, saving on typing: cat data.csv | sed 's/"//g' can be simplified by doing this instead: cat data.csv | tr d '"' This awk command: cat sales.csv | awk -F',' '{print $1}' | sort | uniq Can be replaced with a simpler (IMO) cut instead: cat sales.csv | cut -d , -f 1 | sort | uniq When using head or tail like this: head -n 3 You don't need the -n: head…

I like cut and tr too, but I try to replace them by sed and awk when I can. I reduces the number of moving parts, and allows you to increase the complexity slowly.

Ex: | sed -e step1 becomes | sed -e step1 -e step2 instead of adding another pipe and another "moving part" like tr

Re: An introduction to data processing on the Linux command line

#57

Post by dvrx » Sat, Nov 23, 2019, 9:33 PM UTC

pedro84 wrote: post #29

This is a little more awk-ish: awk -F, '$2 == "F" {$0=(($1-32)*5/9)",C"} {print}'

I love awk too but most people don't know much of awk. Better use regular things and keep awk for whenever you absolutely need it.

Re: An introduction to data processing on the Linux command line

#58

Post by dvrx » Sat, Nov 23, 2019, 9:36 PM UTC

bitminer wrote: post #54

robertelder wrote: post #10

Hi, (I wrote the article). A few people commented noting that I included "Data Science" in the title, but the content doesn't include any statistics or machine learning which is closer to the core definition of 'data science'. I still think the title is appropriate since any kind of low-fidelity data science task you do on some had-hoc data (log files, heaps of text, web pages) is going to start with setting up a pro…

Much can be done just with awk. My pet peeve is the "grep | awk" idiom. No, just use awk. Awk does map/reduce, relational joins, associative memory, table lookup, and so on. Just use awk arrays, begin block, and end block.

If I am going to maintain the script myself and never tweak anything in the middle of the night, sure.

But most people don't know awk. And awk requires more awareness. I break my awk when I fix things when tired.

Re: An introduction to data processing on the Linux command line

#59

Post by clarry » Sat, Nov 23, 2019, 9:49 PM UTC

School-Cotton wrote: post #46

clarry wrote: post #37

Earlier quoted context omitted.

> Why people use Linux in place of nix ? I find it more irritating when people try to score greybeard points by saying *nix (or Unix) when it's obvious that they're talking about a Linux-only mechanism and quite possibly haven't ever used Unix (or a direct derivative).

Huh? What is specific to Linux in this post? Also, the most popular Unix-like OS (far more than Linux) is macOS, basically the least “leet greybeard affectation” thing I can imagine. Your irritation is way off base.

> Huh? What is specific to Linux in this post?

Perhaps nothing? I was responding to the complaint in general terms.

> Your irritation is way off base.

Please allow me to feel irritated when people refer to obvious Linux things as something that's supposedly got something to do with Unix. It happens often enough.

Re: An introduction to data processing on the Linux command line

#60

Post by gnufx » Sat, Nov 23, 2019, 10:17 PM UTC

ibern wrote: post #33

Here are some ways you could simplify some of the tasks in the article, saving on typing: cat data.csv | sed 's/"//g' can be simplified by doing this instead: cat data.csv | tr d '"' This awk command: cat sales.csv | awk -F',' '{print $1}' | sort | uniq Can be replaced with a simpler (IMO) cut instead: cat sales.csv | cut -d , -f 1 | sort | uniq When using head or tail like this: head -n 3 You don't need the -n: head…

If you want to simplify things, don't employ "useless us of cat". Pass the file as a command arg or re-direct input. And sort has options, so the third/fourth commands can be

sort -u -t, sales.csv

However, those fail with quoted commas.

Also, head -3 is non-POSIX obsolete syntax.

Edit: I don't know why I didn't see other UUOC references initially.