Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell Magic: Set Operations with uniq

blog.deadvax.net

Post reply on HN
Print view View original

31–40 of 67 posts

Re: Shell Magic: Set Operations with uniq

#31

Post by int0x80 »

jph wrote: post #19

I like shell set operations scripts, because they are quick and easy. I prefer `awk` over `uniq` and `comm` because awk tends to be faster at set ops that can skip sorting and deduplicating. Here's my script for union, intersection, etc. See README on GitHub. Suggestions welcome. https://github.com/sixarm/setop #!/bin/sh set -eu op="$1"; shift case $op in ∪|u|union|or|∨|add|addition|'+'|'|') awk '!seen[$0] {print} {s…

Thats nice, so is the article's examples. But what I always need to lookup is how to make the operations work on especific columns, rather than on whole lines.

I mean without resorting to awk.

Re: Shell Magic: Set Operations with uniq

#32

Post by kqr »

bewuethr wrote: post #30
dceddia wrote: post #23

Earlier quoted context omitted.

I love how, even after using Unix-y systems for probably 10-odd years, I still occasionally stumble upon something like this that refers to a command that I've never even heard of , and when I check (on my Mac, no less), it's right there sitting in /usr/bin. So many useful little utilities. I think the same thing happens to a lesser degree with vim features.

A very boring, but instructive way is to read manuals cover to cover - for example the GNU Coreutils one [0]. This way, you become aware of the existence of a lot of tools (such as comm, which is part of the coreutils), and while reading you'll realize that some of them might be a better way of doing things than how you're doing them currently. Other boring, but instructive reads (heavily GNU biased): diffutils [1],…

The chapter on shell utilities in the POSIX standard is another goldmine!

Re: Shell Magic: Set Operations with uniq

#33

Post by raverbashing »

sciurus wrote: post #2

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list

comm is probably the most useful command I never remember how it is called (and apropos doesn't help).

Re: Shell Magic: Set Operations with uniq

#35

Post by unixhero »

fiddlerwoaroof wrote: post #28
BenjiWiebe wrote: post #16

Earlier quoted context omitted.

And for people like me who want the file name first on the line, this format works in Bash at least: outfile

I always find it really confusing when the command-line begins with a redirect.

Yes, how do you parse that - ie. what does it even mean?

Re: Shell Magic: Set Operations with uniq

#36

Post by therein »

TheDong wrote: post #12
badminton1 wrote: post #7

Earlier quoted context omitted.

Can also use comm To use the output of a command as input. This also works with diff and other commands.

More accurately, it can be used for any command which expects a file and doesn't do anything too weird in reading it (e.g. doesn't seek to the beginning and read it again) The ' $ ls

Thank you. I was just thinking of looking into how it was implemented.

Re: Shell Magic: Set Operations with uniq

#37

Post by fiddlerwoaroof »

unixhero wrote: post #35
fiddlerwoaroof wrote: post #28

Earlier quoted context omitted.

I always find it really confusing when the command-line begins with a redirect.

Yes, how do you parse that - ie. what does it even mean?

It's just a convention that redirects come after the command they're redirecting. There might be shell options that influence this, but I think all these are generally equivalent:

    foo bar

Re: Shell Magic: Set Operations with uniq

#39

Post by kseistrup »

pixelbeat__ wrote: post #9

I've a summary of the operations at: http://www.pixelbeat.org/cmdline.html#sets Note comm output is a bit awkward to parse, so I use another coreutils `join` command to process already sorted data

Thanks for the list, there are many useful examples.

Another way to get the last date in current month (== the number of days in a month) is:

    : $(cal); echo $_

Re: Shell Magic: Set Operations with uniq

#40

Post by unixhero »

jph wrote: post #19

I like shell set operations scripts, because they are quick and easy. I prefer `awk` over `uniq` and `comm` because awk tends to be faster at set ops that can skip sorting and deduplicating. Here's my script for union, intersection, etc. See README on GitHub. Suggestions welcome. https://github.com/sixarm/setop #!/bin/sh set -eu op="$1"; shift case $op in ∪|u|union|or|∨|add|addition|'+'|'|') awk '!seen[$0] {print} {s…

A few comments. First, wow, I love how this looks like dark magic.

Those mathematical notations, are you using them because it makes it easier to see how it corresponds to actual Set Theory/theorems? If so, could you just as well have used an alphanummeric identifier like "left" "union" "right" or - would the code break without this notation? I'm on deep waters here, I don't know this. But set theory seems to pop up a lot in my line of work, essentially doing joins in datasets using Tableau - so my interest in the nitty gritty of this field is increasing.

Post reply on HN