Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Shell Magic: Set Operations with uniq

blog.deadvax.net

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Shell Magic: Set Operations with uniq

#61

Post by jph »

fiddlerwoaroof wrote: post #27
jph wrote: post #19

I like shell set operations scripts, because they are quick and easy. I prefer `awk` over `uniq` and `comm` because awk tends to be faster at set ops that can skip sorting and deduplicating. Here's my script for union, intersection, etc. See README on GitHub. Suggestions welcome. https://github.com/sixarm/setop #!/bin/sh set -eu op="$1"; shift case $op in ∪|u|union|or|∨|add|addition|'+'|'|') awk '!seen[$0] {print} {s…

doesn't this read the entire input into memory? `uniq` and `comm` don't (need to) do this, so they can work on inputs bigger than available memory.

Yes in the `setop` current implementation, because this enables the script to be fast, short, and work with inputs without needing to call `sort`.

For comparison, a typical POSIX `uniq` implementation reads the input and solely compares two adjacent lines; this requires the input to be presorted.

An interesting upgrade could be to add a `setop` option flag that tells the script the inputs are already sorted and/or deduped. This can achieve the memory savings you're describing.

Re: Shell Magic: Set Operations with uniq

#63

Post by Anthony-G »

sciurus wrote: post #59
bewuethr wrote: post #30

Earlier quoted context omitted.

A very boring, but instructive way is to read manuals cover to cover - for example the GNU Coreutils one [0]. This way, you become aware of the existence of a lot of tools (such as comm, which is part of the coreutils), and while reading you'll realize that some of them might be a better way of doing things than how you're doing them currently. Other boring, but instructive reads (heavily GNU biased): diffutils [1],…

I'd also recommend Unix Power Tools. There's definitely some outdated content, but I learned a lot about shell usage and text manipulation from it. http://shop.oreilly.com/product/9780596003302.do

Seconded. The dead tree copy I purchased in the mid-2000s is the best technical book in my collection. I’ve read most of it and every so often I still dip in to learn something new or – more often than not – to re-learn something I’ve forgotten.

Re: Shell Magic: Set Operations with uniq

#64

Post by FractalLP »

jph wrote: post #25
FractalLP wrote: post #24

Earlier quoted context omitted.

How long have you been using Awk?

For a few years for two clients (financial and governmental) that have a range of older controlled systems. We benefit from simple analytics that are POSIX, not python, perl, R, etc. http://www.numcommand.com/

I've never heard of numcommand before, but it looks super awesome and exactly what I need. I use Linux servers, but am forced to use Windows as my local. So first question, do you have experience with awk/gawk on Windows? Second question: What do you prefer about Awk over Python/Perl? I guess if all your analytics are fairly short it is just easier in Awk? Thanks!

Re: Shell Magic: Set Operations with uniq

#65

Post by jph »

FractalLP wrote: post #64
jph wrote: post #25

Earlier quoted context omitted.

For a few years for two clients (financial and governmental) that have a range of older controlled systems. We benefit from simple analytics that are POSIX, not python, perl, R, etc. http://www.numcommand.com/

I've never heard of numcommand before, but it looks super awesome and exactly what I need. I use Linux servers, but am forced to use Windows as my local. So first question, do you have experience with awk/gawk on Windows? Second question: What do you prefer about Awk over Python/Perl? I guess if all your analytics are fairly short it is just easier in Awk? Thanks!

Thank you!

Yes in my experience of awk/gawk on Windows, things work the same, other than file separators, line endings, and similar platform-specific issues.

For your second question, awk simply works. It's reliable, fast, and everywhere. I also like python and perl, and choose these for any modern system.

Re: Shell Magic: Set Operations with uniq

#67

Post by FractalLP »

jph wrote: post #65
FractalLP wrote: post #64

Earlier quoted context omitted.

I've never heard of numcommand before, but it looks super awesome and exactly what I need. I use Linux servers, but am forced to use Windows as my local. So first question, do you have experience with awk/gawk on Windows? Second question: What do you prefer about Awk over Python/Perl? I guess if all your analytics are fairly short it is just easier in Awk? Thanks!

Thank you! Yes in my experience of awk/gawk on Windows, things work the same, other than file separators, line endings, and similar platform-specific issues. For your second question, awk simply works. It's reliable, fast, and everywhere. I also like python and perl, and choose these for any modern system.

Thank you!
Post reply on HN