1) `pgrep` is a standard utility that does what his `psgrep` does and much much more. 2) htop is a cpu and memory hog -- every time I've used it I noticed it takes 6+% CPU time 3) there's an awk trick to do the `sort | uniq` recommendation that works on 10+GB files (single pass): awk '!x[$0]++' 4) Passwordless keys are dangerous -- use ssh-agent to save the password of the keys
3) if all the lines of the 10+GB file are actually unique, wouldn't awk keep the whole file in RAM? For files larger than my RAM could this leave my system unresponsive because it's thrashing on swap?
For files with relatively few duplicates it's going to be a lot slower than sort | uniq.
Trying it on a 128MB file (nowhere near enough time to test a 10GB file) filled with lines of 7 random upper case characters[1] (so hardly any duplicates):-
$ wc -l x.out
16777216 x.out
$ time ( sort x.out | uniq ) | wc -l
16759719
real 0m17.982s
user 0m42.575s
sys 0m0.876s
$ time ( sort -u x.out ) | wc -l
16759719
real 0m20.582s
user 0m43.775s
sys 0m0.688s
As for the awk method:-
$ time awk '!x[$0]++' x.out | wc -l
This Linux machine has ~16GB of memory so the file was going to be completed cached in memory before the first test. All things considered equal the awk method will be roughly O(n) (e.g. linear against file size) and sort/uniq will be O(n log n). So, theoretically, the awk method will eventually surpass the sort method because it's having to do less work (it's only checking for a previously seen key rather than sorting the entire file) but I'm not sure the crossover will be anywhere useful if the file doesn't contain many duplicates.
Repeating it for a file containing lots of duplicates (same 128MB file size but contents are only the 7 letter words consisting of A or B, so only 128 possible entries):-
$ time awk '!x[$0]++' y.out | wc -l
128
real 0m1.207s
user 0m1.192s
sys 0m0.016s
$ time ( sort y.out | uniq ) | wc -l
128
real 0m14.320s
user 0m31.414s
sys 0m0.428s
$ time ( sort -u y.out | uniq ) | wc -l
128
real 0m12.638s
user 0m30.366s
sys 0m0.188s
So awk is much faster for files with lots of duplicates. No great surprises. When I get a chance I'll repeat it for a 1GB file and a 10GB file (with lots of duplicates otherwise the awk version will take far too long).
1. Example contents:-
EPQKHPH DLJCROB WICVGQY MHWTPSR HMPNECN