Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell Magic: Set Operations with uniq

blog.deadvax.net

Post reply on HN
Print view View original

11–20 of 67 posts

Re: Shell Magic: Set Operations with uniq

#11

Post by buddydvd »

sciurus wrote: post #2

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list

I tried out your command and got different line count compared to the output of the commands mentioned in the post. The man page for comm says it assumes inputs to be pre-sorted. To do that, you need to sort the input files before passing them to comm.

  # show only items in both a and b
  comm -1 -2

Re: Shell Magic: Set Operations with uniq

#12

Post by TheDong »

badminton1 wrote: post #7
sciurus wrote: post #2

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list

Can also use comm To use the output of a command as input. This also works with diff and other commands.

More accurately, it can be used for any command which expects a file and doesn't do anything too weird in reading it (e.g. doesn't seek to the beginning and read it again)

The '

    $ ls

Re: Shell Magic: Set Operations with uniq

#13

Post by scintill76 »

badminton1 wrote: post #7
sciurus wrote: post #2

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list

Can also use comm To use the output of a command as input. This also works with diff and other commands.

Note that this is a bash feature; other shells may have different syntax.

Re: Shell Magic: Set Operations with uniq

#15

Post by alpb »

sciurus wrote: post #2

This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list

Indeed comm will work much faster than uniq in this case, because it reads each file only once, and does a single pass on each file.

What the blog post does (doing cat multiple times, and then counting the occurrences) is going to be much slower, although mathematically correct.

Re: Shell Magic: Set Operations with uniq

#16

Post by BenjiWiebe »

kbob wrote: post #14

Aaargh! "cat | sort" is an antipattern. Sort accepts a list of files as arguments. "sort [flags] file1 file2 ..." is more concise, more efficient, and is less likely to run out of space.

And for people like me who want the file name first on the line, this format works in Bash at least:

outfile

Re: Shell Magic: Set Operations with uniq

#17

Post by mds »

kbob wrote: post #14

Aaargh! "cat | sort" is an antipattern. Sort accepts a list of files as arguments. "sort [flags] file1 file2 ..." is more concise, more efficient, and is less likely to run out of space.

Someone always says this, but starting with cat sure makes it easier compose the command line. When working with big files, start with "head -1000 | sort | uniq ..." and switch to cat after you work out the command. Or you start composing the command and realize you need to sort on field 2, so insert a "cut" before the "sort" etc.

Re: Shell Magic: Set Operations with uniq

#19

Post by jph »

I like shell set operations scripts, because they are quick and easy.

I prefer `awk` over `uniq` and `comm` because awk tends to be faster at set ops that can skip sorting and deduplicating.

Here's my script for union, intersection, etc. See README on GitHub. Suggestions welcome.

https://github.com/sixarm/setop

    #!/bin/sh
    set -eu
    op="$1"; shift

    case  $op  in
      ∪|u|union|or|∨|add|addition|'+'|'|')
        awk '!seen[$0] {print} {seen[$0]=1}' "$@"
        ;;
      ∩|i|intersection|and|∧|'&')
        awk 'FNR==1{argind+=1} seen[$0]+=1 {next} END { for (key in seen) { if (seen[key]==argind) { print key } } }' "$@"
        ;;
      ⊖|d|diff|difference|xor|⊻)
        awk 'FNR==1{argind+=1} seen[$0]+=1 {next} END { for (key in seen) { if (seen[key]==1) { print key } } }' "$@"
        ;;
      ex|except|exclude|subtract|subtraction|'-')
        awk 'NR==FNR{seen[$0]=1;next} seen[$0]=0; END { for (key in seen) { if (seen[key]) { print key } } }' "$@"
        ;;
      extra)
        awk 'BEGIN{argindmax=ARGC-1} FNR==1{argind+=1} argind

Re: Shell Magic: Set Operations with uniq

#20

Post by perl4ever »

derriz wrote: post #4

Hard to resist this - "Unlike the intersection, the Set Difference is a bit harder to scale up to more than two lists. It is concievable, and I may even have done it, but I’ll leave it as an exercise to the reader to develop that." An inefficient solution which involves unnecessary sorting: for each file_i, 0 <= i < n in the set of n files, cat it 2^i times before combining to pipe through sort and uniq -c. Every pos…

The way I would do it, assuming we have list1.txt, list2.txt, list3.txt, ... and want to calculate (1 - 2 - 3 - ...) :

1. Use sed to add "1" (that's a one digit and a tab char) to the first list to difference by and save as "prefix.txt".

2. Use cat to combine all the lists, sort | uniq -c | sort -n | sed | sort again and save as output.txt.

3. join prefix.txt and output.txt on each whole line and cut the second tab delimited field to produce the final result.

So in order to be in the result, a list item must appear in exactly one list and that must be the first list. That should be what we want (?)

Post reply on HN