Useful Unix commands for exploring data
datavu.blogspot.com
Useful Unix commands for exploring data
1–10 of 156 posts
Re: Useful Unix commands for exploring data
#2caveat: delimiter-based commands are not quote-aware. For example, this is a CSV line with two fields:
foo,"bar,baz"
$ echo 'foo,"bar,baz"' | awk -F, '{print NF}'
3Re: Useful Unix commands for exploring data
#3>> If we don't want new file we can redirect the output to same file which will overwrite original file
You need to be a little careful with that. If you do:
uniq -u movies.csv > movies.csv
Of course when uniq opens movies.csv for consumption, it'll already be empty. There will be no work to do.
There's a couple of options to deal with this, but the temporary intermediate file is my preference provided there's sufficient space - it's easily understood, if someone else comes across the construct in your script, they'll grok it.
Re: Useful Unix commands for exploring data
#4uniq also doesn't deal well with duplicate records that aren't adjacent. You may need to do a sort before using it.
sort | uniq
Re: Useful Unix commands for exploring data
#5uniq also doesn't deal well with duplicate records that aren't adjacent. You may need to do a sort before using it. sort | uniq But that can screw with your header lines, so be careful there two.
You can do this without sorting:
awk '!x[$0]++'Re: Useful Unix commands for exploring data
#6caveat: delimiter-based commands are not quote-aware. For example, this is a CSV line with two fields: foo,"bar,baz" However, the tools will treat it as 3 columns: $ echo 'foo,"bar,baz"' | awk -F, '{print NF}' 3
Is there any workaround?
Re: Useful Unix commands for exploring data
#7>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There…
I came here for that. I learned long ago, the hard way, to never ever use the same file for writing as reading. I was wondering if that rule had changed on me.
Re: Useful Unix commands for exploring data
#8uniq also doesn't deal well with duplicate records that aren't adjacent. You may need to do a sort before using it. sort | uniq But that can screw with your header lines, so be careful there two.
sort -u
Re: Useful Unix commands for exploring data
#9caveat: delimiter-based commands are not quote-aware. For example, this is a CSV line with two fields: foo,"bar,baz" However, the tools will treat it as 3 columns: $ echo 'foo,"bar,baz"' | awk -F, '{print NF}' 3
Is there any workaround?
Practically all Unix tools consider the comma-separated-but-you-can-use-quotes-to-override CSV file to be an abomination. [1] You have to have a crazy regexp to get around it.
[1] Maybe someday they won't.
Re: Useful Unix commands for exploring data
#10>> If we don't want new file we can redirect the output to same file which will overwrite original file You need to be a little careful with that. If you do: uniq -u movies.csv > movies.csv The shell will first open movies.csv for writing (the redirect part) then launch the uniq command connecting stdout to the now emptied movies.csv. Of course when uniq opens movies.csv for consumption, it'll already be empty. There…
Thank you for inputs, how about this?
uniq -u movies.csv > temp.csv
temp.csv > movie.csv
rm temp.csv