Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell Magic: Set Operations with uniq

blog.deadvax.net

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Shell Magic: Set Operations with uniq

#51

Post by nonsince »

For completeness, to scale relative complement up to N lists you would need to repeat the last list 2^N times. So it'd look something like:

    rel_com() {
        for i in $(seq 1 $#); do
            for j in $(seq 1 $(bc
Then you have a N-bit number with the i'th bit representing membership in the i'th file.

Re: Shell Magic: Set Operations with uniq

#52

Post by kseistrup »

lloeki wrote: post #48
kseistrup wrote: post #39

Earlier quoted context omitted.

Thanks for the list, there are many useful examples. Another way to get the last date in current month (== the number of days in a month) is: : $(cal); echo $_

For the curious, `:` is the noop builtin in bash. I use it mostly as I would use `pass` in python, since empty conditionals or functions are a parse error. if whatevs; then : fi

And to elaborate even more:

‘:’ is the command, and ‘$(cal)’ – which equals the unquoted output from running the ‘cal’ command – are the arguments. The last day of the month is thus the last argument of ‘:’ and can be referenced with the ‘$_’ variable.

Re: Shell Magic: Set Operations with uniq

#54

Post by paxunix »

raverbashing wrote: post #33
sciurus wrote: post #2

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list

comm is probably the most useful command I never remember how it is called (and apropos doesn't help).

The only options you usually need for comm are -1, -2, and -3. It takes two arguments, call them 1 and 2. Its job is to include lines from each and both files in output (by default, you get 3 columns of output), so we want to filter by telling it to exclude what we don't want. My mnemonic is "-" means "not", so -1 means "not only in file 1", -2 means "not only in file 2", and -3 means "not in both". So now, when I want comm output that shows the lines unique to my 2nd file, it's:

comm "not only in 1" and "not in both"

so:

comm -1 -3 file1 file2

Re: Shell Magic: Set Operations with uniq

#55

Post by paxunix »

fiddlerwoaroof wrote: post #37
unixhero wrote: post #35

Earlier quoted context omitted.

Yes, how do you parse that - ie. what does it even mean?

It's just a convention that redirects come after the command they're redirecting. There might be shell options that influence this, but I think all these are generally equivalent: foo bar

It is POSIX shell parsing behaviour that redirections can appear anywhere in the command (obviously, not in the same word as another parameter, nor inside a quoted string). They have to be stripped out by the shell before execution: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...

Re: Shell Magic: Set Operations with uniq

#56

Post by alpb »

fiddlerwoaroof wrote: post #26
alpb wrote: post #15

Earlier quoted context omitted.

Indeed comm will work much faster than uniq in this case, because it reads each file only once, and does a single pass on each file. What the blog post does (doing cat multiple times, and then counting the occurrences) is going to be much slower, although mathematically correct.

Doesn't uniq only read the input once? I've always assumed that the reason uniq assumes its input is presorted is that, then, it doesn't need to buffer all of it: instead, it just eliminates successive duplicate lines.

Yeah but in this case the author is doing "cat 1.txt 1.txt 2.txt".

Re: Shell Magic: Set Operations with uniq

#57

Post by raverbashing »

paxunix wrote: post #54
raverbashing wrote: post #33

Earlier quoted context omitted.

comm is probably the most useful command I never remember how it is called (and apropos doesn't help).

The only options you usually need for comm are -1, -2, and -3. It takes two arguments, call them 1 and 2. Its job is to include lines from each and both files in output (by default, you get 3 columns of output), so we want to filter by telling it to exclude what we don't want. My mnemonic is "-" means "not", so -1 means "not only in file 1", -2 means "not only in file 2", and -3 means "not in both". So now, when I wa…

I know how to use 'man', what I mean is, I never remember the command name (comm) because I always remember it's a command to compare so I think of cmp, and not comm.

Re: Shell Magic: Set Operations with uniq

#58

Post by sciurus »

buddydvd wrote: post #11
sciurus wrote: post #2

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list

I tried out your command and got different line count compared to the output of the commands mentioned in the post. The man page for comm says it assumes inputs to be pre-sorted. To do that, you need to sort the input files before passing them to comm. # show only items in both a and b comm -1 -2

Yes, pre-sorting is necessary for comm to work. I didn't mention that because it's necessary for the commands in the post to work too.

Re: Shell Magic: Set Operations with uniq

#59

Post by sciurus »

bewuethr wrote: post #30
dceddia wrote: post #23

Earlier quoted context omitted.

I love how, even after using Unix-y systems for probably 10-odd years, I still occasionally stumble upon something like this that refers to a command that I've never even heard of , and when I check (on my Mac, no less), it's right there sitting in /usr/bin. So many useful little utilities. I think the same thing happens to a lesser degree with vim features.

A very boring, but instructive way is to read manuals cover to cover - for example the GNU Coreutils one [0]. This way, you become aware of the existence of a lot of tools (such as comm, which is part of the coreutils), and while reading you'll realize that some of them might be a better way of doing things than how you're doing them currently. Other boring, but instructive reads (heavily GNU biased): diffutils [1],…

I'd also recommend Unix Power Tools. There's definitely some outdated content, but I learned a lot about shell usage and text manipulation from it.

http://shop.oreilly.com/product/9780596003302.do

Re: Shell Magic: Set Operations with uniq

#60

Post by jph »

unixhero wrote: post #40
jph wrote: post #19

I like shell set operations scripts, because they are quick and easy. I prefer `awk` over `uniq` and `comm` because awk tends to be faster at set ops that can skip sorting and deduplicating. Here's my script for union, intersection, etc. See README on GitHub. Suggestions welcome. https://github.com/sixarm/setop #!/bin/sh set -eu op="$1"; shift case $op in ∪|u|union|or|∨|add|addition|'+'|'|') awk '!seen[$0] {print} {s…

A few comments. First, wow, I love how this looks like dark magic. Those mathematical notations, are you using them because it makes it easier to see how it corresponds to actual Set Theory/theorems? If so, could you just as well have used an alphanummeric identifier like "left" "union" "right" or - would the code break without this notation? I'm on deep waters here, I don't know this. But set theory seems to pop up…

Thanks! To answer your questions...

> because it makes it easier to see how it corresponds to actual Set Theory/theorems?

Yes. These are the Unicode symbols for set theory.

> could you just as well have used an alphanummeric identifier like "left" "union" "right" or

Yes. You can use any of the words in the case switch statements, such as `setop union file1 file2`. You can also edit the script to add your own words if you like.

You can see simpler versions of these scripts in our GitHub repos. For example the `union` command is https://github.com/sixarm/union

> set theory seems to pop up a lot in my line of work

More and more in mine too. Thank you for your comments!

Post reply on HN