21–30 of 156 posts
uniq also doesn't deal well with duplicate records that aren't adjacent. You may need to do a sort before using it. sort | uniq But that can screw with your header lines, so be careful there two.
You can do this without sorting: awk '!x[$0]++'
awk '!($0 in a);a[$0]; print}'
I rarely if ever use uniq to remove duplicates. Sorting is expensive.
>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There…
http://linux.die.net/man/1/sponge
uniq -u movies.csv | sponge movies.csvcaveat: delimiter-based commands are not quote-aware. For example, this is a CSV line with two fields: foo,"bar,baz" However, the tools will treat it as 3 columns: $ echo 'foo,"bar,baz"' | awk -F, '{print NF}' 3
Is there any workaround?
You can write one-off (or reuseable) filters in minutes.
lex/flex should be in every UNIX distribution that has a C compiler, but maybe that's changing.
I love Unix pipelines, but chances are your data is structured in such a way that using regex based tools will break that structure unless you're very careful. You know that thing about not making HTML with regexs? Same rule applies to CSV, TSV, and XLSX. All these can be created, manipulated and read using Python, which is probably already on your system.
I've used gawk for many things ranging from data analysis to generate linker / loader code in an embedded build environment for a custom processor / sequencer.
(You can even find a version to run from the Windows command prompt if you don't have Cygwin.)
>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There…
The utility to do this is called sponge. http://linux.die.net/man/1/sponge uniq -u movies.csv | sponge movies.csv
Note: ’uniq’ does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use ‘sort -u’ without ‘uniq’.
Your movies.csv file is already sorted, but you don't mention that sorting is important for using uniq, which may be misleading.
$ cat tmp.txt
AAAA
AAAA
BBBB
DDDD
BBBB
$ uniq -d tmp.txt
AAAA
>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There…
The utility to do this is called sponge. http://linux.die.net/man/1/sponge uniq -u movies.csv | sponge movies.csv
uniq also doesn't deal well with duplicate records that aren't adjacent. You may need to do a sort before using it. sort | uniq But that can screw with your header lines, so be careful there two.
F=filename; (head -n 1 $F ; tail -n +2 $F | sort -u) | sponge $F
sort filename | uniq -c | awk '$1 != 1'