How does Awk ' a[$0]++' work?
unix.stackexchange.com
How does Awk ' a[$0]++' work?
1–9 of 9 posts
Re: How does Awk ' a[$0]++' work?
#2Ahh, the (apparently?) lost art of command line voodoo. This was type of "write once, read never again" culture. Perl was a similar guy in this family, and if you really got good at it, you could of course read again these gems and produce them, too.
This is quite a different culture then "young folks" [TM] who write a unit-tested nodejs library with tons of dependencies to do the same ;-)
Of course I'm exaggerating, but there's some wisedom in these onliners, or Perl5 with it's sigils.
Re: How does Awk ' a[$0]++' work?
#3Ahh, the (apparently?) lost art of command line voodoo. This was type of "write once, read never again" culture. Perl was a similar guy in this family, and if you really got good at it, you could of course read again these gems and produce them, too. This is quite a different culture then "young folks" [TM] who write a unit-tested nodejs library with tons of dependencies to do the same ;-) Of course I'm exaggerating,…
I wouldn't say it's a lost art, but perhaps declining in ubiquity. I'm an SRE; CLI wizardry is part and parcel of the job. jq is especially helpful/powerful for various services that output JSON.
Re: How does Awk ' a[$0]++' work?
#4There is a "!" missing from the title[0], should be
awk '!a[$0]++'
Also, this version[1], which works with gawks from 2013 and after, changes the file in place as it goes, where the original version just makes the array which then needs to be printed/saved:
gawk -i inplace '!a[$0]++' file
p.s. The current misspelled title using "Awk", raises another question–I was surprised that "Awk '!a[$0]++'" works too (on Mac at least). If I "ls Awk" in \usr\bin, it says Awk is there. If I "ls awk" it says awk is also there—but it seems they're the same file, and it's only pretending Awk is a file. AWK also... I never noticed that before!
[0] I sympathize—I once tried to submit a !!Con talk.
Re: How does Awk ' a[$0]++' work?
#5Ahh, the (apparently?) lost art of command line voodoo. This was type of "write once, read never again" culture. Perl was a similar guy in this family, and if you really got good at it, you could of course read again these gems and produce them, too. This is quite a different culture then "young folks" [TM] who write a unit-tested nodejs library with tons of dependencies to do the same ;-) Of course I'm exaggerating,…
I wouldn't say it's a lost art, but perhaps declining in ubiquity. I'm an SRE; CLI wizardry is part and parcel of the job. jq is especially helpful/powerful for various services that output JSON.
This has haunted me working in the infrastructure automation space. It is utterly dominated by wizards quickly solving problems with one-off scripts and then committing those to Kubernetes CRDs embedded as bash script heredocs in a yaml list that are completely incomprehensible to anyone else trying to read them to figure out how a system works. It means coming onto teams that are completely dominated by whoever happened to work there first because they wrote all the custom tooling and nobody else understands it, leaving us with fires all the time where everyone else can scramble for hours or days to figure out what the heck is happening, but he shows up and fixes it in ten seconds because he wrote the bad code and immediately knows what the problem is.
Re: How does Awk ' a[$0]++' work?
#6There is a "!" missing from the title[0], should be awk '!a[$0]++' "removes duplicates while maintaining order. a feat unachieved by uniq or sort -u which either removes only adjacent duplicates or has to break order to remove duplicates." Also, this version[1], which works with gawks from 2013 and after, changes the file in place as it goes, where the original version just makes the array which then needs to be prin…
As for "awk", "Awk", and "AWK" all working on macOS: APFS seems to default to being case-insensitive.
Re: How does Awk ' a[$0]++' work?
#7There is a "!" missing from the title[0], should be awk '!a[$0]++' "removes duplicates while maintaining order. a feat unachieved by uniq or sort -u which either removes only adjacent duplicates or has to break order to remove duplicates." Also, this version[1], which works with gawks from 2013 and after, changes the file in place as it goes, where the original version just makes the array which then needs to be prin…
> I was surprised that "Awk '!a[$0]++'" works too (on Mac at least). If I "ls Awk" in \usr\bin, it says Awk is there. If I "ls awk" it says awk is also there—but it seems they're the same file, and it's only pretending Awk is a file. AWK also... I never noticed that before!
I presume this is a side-effect of the macOS default filesystem being case-insensitive. I'm running macOS with a case-sensitive fs and that does not work:
$ awk
usage: awk [-F fs] [-v var=value] [-f progfile | 'prog'] [file ...]
$ Awk
-bash: Awk: command not found
Re: How does Awk ' a[$0]++' work?
#8There is a "!" missing from the title[0], should be awk '!a[$0]++' "removes duplicates while maintaining order. a feat unachieved by uniq or sort -u which either removes only adjacent duplicates or has to break order to remove duplicates." Also, this version[1], which works with gawks from 2013 and after, changes the file in place as it goes, where the original version just makes the array which then needs to be prin…
This explains how "-i inplace" works: https://www.gnu.org/software/gawk/manual/html_node/Extension...
Re: How does Awk ' a[$0]++' work?
#9Earlier quoted context omitted.
I wouldn't say it's a lost art, but perhaps declining in ubiquity. I'm an SRE; CLI wizardry is part and parcel of the job. jq is especially helpful/powerful for various services that output JSON.
This has haunted me working in the infrastructure automation space. It is utterly dominated by wizards quickly solving problems with one-off scripts and then committing those to Kubernetes CRDs embedded as bash script heredocs in a yaml list that are completely incomprehensible to anyone else trying to read them to figure out how a system works. It means coming onto teams that are completely dominated by whoever happ…
Interesting -- what part of this infrastructure would you like to change? The wizards would probably want to get rid of YAML, probably even of Kubernetes and just run their scripts on bare metal. The DevOps would probably want to get rid of the bash scripts in favour of some "sane" tool written in a language which allows for testing, extension, etc.
This is one of these "it works for me" problems, the main issue is probably actually the lack of documentation. I wonder why quickly rushing wizards don't adopt the dictation technique used by (alway rushing) medicals/physicians. They outsource the transcription to lower paid assistants.