Shell Magic: Set Operations with uniq
blog.deadvax.net
Shell Magic: Set Operations with uniq
1–10 of 67 posts
Re: Shell Magic: Set Operations with uniq
#2This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm.
https://en.wikipedia.org/wiki/Comm
# show only items in both a and b
comm -1 -2 a_list b_list
# show only items unique to a
comm -2 -3 a_list b_list
# show only items unique to b
comm -1 -3 a_list b_listRe: Shell Magic: Set Operations with uniq
#3Thanks! Will be using these for debugging. I commonly compare `cat data.csv | wc -l` vs `cat data.csv | sort -u | wc -l` to check unique inputs
Re: Shell Magic: Set Operations with uniq
#4Hard to resist this - "Unlike the intersection, the Set Difference is a bit harder to scale up to more than two lists. It is concievable, and I may even have done it, but I’ll leave it as an exercise to the reader to develop that."
An inefficient solution which involves unnecessary sorting: for each file_i, 0 <= i < n in the set of n files, cat it 2^i times before combining to pipe through sort and uniq -c. Every possible set operation combination can be determined by grepping the result for a particular combination of counts. Intersection would calculated by grepping for 2^n - 1 while symmetric difference would require egrep to pick out any of 1, 2, 4,..,2^(n-1).
Re: Shell Magic: Set Operations with uniq
#5I've always used grep for this with the -f option to read a list of patterns from a file and -v for inverted search as necessary.
Re: Shell Magic: Set Operations with uniq
#6The cut and join commands are also useful. I'm pretty sure you could prove all basic database operations can be done using the shell.
Re: Shell Magic: Set Operations with uniq
#7This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list
Can also use
comm Re: Shell Magic: Set Operations with uniq
#8This is a neat hack! For two input files, the intersection and relative complement can be done more straightforwardly with comm. https://en.wikipedia.org/wiki/Comm # show only items in both a and b comm -1 -2 a_list b_list # show only items unique to a comm -2 -3 a_list b_list # show only items unique to b comm -1 -3 a_list b_list
[deleted]
Re: Shell Magic: Set Operations with uniq
#9I've a summary of the operations at:
http://www.pixelbeat.org/cmdline.html#sets
Note comm output is a bit awkward to parse, so I use another coreutils `join` command to process already sorted data
Re: Shell Magic: Set Operations with uniq
#10The cut and join commands are also useful. I'm pretty sure you could prove all basic database operations can be done using the shell.
In fact I wrote an image/tag database solution entirely in shell/coreutils.
It's too disgusting to share.