Live data from Hacker News

Shell Magic: Set Operations with uniq

blog.deadvax.net

41–50 of 67 posts

Re: Shell Magic: Set Operations with uniq

#41
post #25

Earlier quoted context omitted.

How long have you been using Awk?

For a few years for two clients (financial and governmental) that have a range of older controlled systems. We benefit from simple analytics that are POSIX, not python, perl, R, etc. http://www.numcommand.com/

'Hat tip

Re: Shell Magic: Set Operations with uniq

#45
post #19

I like shell set operations scripts, because they are quick and easy. I prefer `awk` over `uniq` and `comm` because awk tends to be faster at set ops that can skip sorting and deduplicating. Here's my script for union, intersection, etc. See README on GitHub. Suggestions welcome. https://github.com/sixarm/setop #!/bin/sh set -eu op="$1"; shift case $op in ∪|u|union|or|∨|add|addition|'+'|'|') awk '!seen[$0] {print} {s…

Could you add a "--help" option? Or provide usage instructions in a comment?

Re: Shell Magic: Set Operations with uniq

#46
If your input is already sorted (like this article assumes), you can use "sort -m", which is a lot faster. Also, to print only lines with duplicates, use "uniq -d" instead of "uniq -c | grep 2\ ".

Union: Instead of

    cat a_list b_list | sort | uniq
do

    sort -m a_list b_list | uniq
Intersection: Instead of

    cat a_list b_list | sort | uniq -c | grep 2\ 
do

    sort -m a_list b_list | uniq -d
Relative complement: Instead of

    cat a_list b_list b_list | sort | uniq -c | grep 2\ 
do

    sort -m a_list a_list b_list | uniq -u
Note the change of approach here: instead of making lines from b_list appear twice and grepping for that count, make lines from a_list appear twice and have uniq only print lines that aren't repeated.

Re: Shell Magic: Set Operations with uniq

#47
post #36
post #12

Earlier quoted context omitted.

More accurately, it can be used for any command which expects a file and doesn't do anything too weird in reading it (e.g. doesn't seek to the beginning and read it again) The ' $ ls

Thank you. I was just thinking of looking into how it was implemented.

It is called "process substitution", and can work the other way around:

http://www.tldp.org/LDP/abs/html/process-sub.html

With fish:

https://fishshell.com/docs/current/commands.html#psub

But there it only works one way:

https://github.com/fish-shell/fish-shell/issues/1786

Re: Shell Magic: Set Operations with uniq

#48

I've a summary of the operations at: http://www.pixelbeat.org/cmdline.html#sets Note comm output is a bit awkward to parse, so I use another coreutils `join` command to process already sorted data

Thanks for the list, there are many useful examples. Another way to get the last date in current month (== the number of days in a month) is: : $(cal); echo $_

For the curious, `:` is the noop builtin in bash. I use it mostly as I would use `pass` in python, since empty conditionals or functions are a parse error.

    if whatevs; then
      :
    fi

Re: Shell Magic: Set Operations with uniq

#49

Earlier quoted context omitted.

Can also use comm To use the output of a command as input. This also works with diff and other commands.

Note that this is a bash feature; other shells may have different syntax.

It's the same in ksh at least, so one is covered in the BSDs. It could be also implemented as a separate tool, but not as convenient, because of additional quoting - I did once.

Re: Shell Magic: Set Operations with uniq

#50
post #2

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list

Can also use comm To use the output of a command as input. This also works with diff and other commands.

I use it in my frugal SSG project [0] to remove HTML files for every removed source file. It's great to use it this way as it doesn't use temp files.

[0] https://gist.github.com/hadrianw/060944011acfcadd889d937b960...

Post reply on HN