Skip grep, use awk

I used Python most of the time but still use Perl where it is appropriate.

You need to be a touch careful with blanket statements, as this conclusion isn't quite what the data in my blog says. What it says is that ag is generally slower than GNU grep on large files, primarily because GNU grep's core search code is quite a bit more optimized than ag's. However, ag can outclass GNU grep when searching across entire directories primarily by culling the set of files that is searched, and also by parallelizing search if you want to exclude running GNU grep in a simple `find ... | xargs -P`-like command. (This is thesmallestcat's point.) This is why the first set of benchmarks in my blog don't actually benchmark GNU grep, because there is an impedance mismatch. Instead, ag/pt are benchmarked against `git grep`, where the comparison is quite a bit closer. (You'll want to check out the benchmarks on my local machine, which avoid penalizing ag for its use of memory maps.[1]) The second set of benchmarks basically swap out `git grep` for GNU grep and compare the tools on very large files.

ack isn't actually included in the benchmarks because it was incredibly slow when I tried it, although there may be something funny going on.[2] To be honest, it isn't terribly surprising to me, since every other tool is compiled down to native code.

The pt/sift story is interesting, and my current hypothesis is that the tools have received a misleading reputation for being fast. In particular, when using the tools with simple literals, it's likely that search will benefit from an AVX2 optimization when searching for the first byte. As soon as you start feeding more complex patterns (including case insensitivity), these tools slow down quite a bit because 1) the underlying regex engine needs more optimization work and 2) the literal analysis is lacking, which means the tools rely even more on the regex engine than other tools do.

The short summary of ripgrep is that it should outclass all the tools on all the benchmarks in my blog. I should update them at some point as ripgrep has gotten a bit faster since that blog post. Primarily, a parallel recursive directory iterator, which is present in sift, pt and ucg as well. Secondarily, its line counting has been sped up with more specialized SIMD routines. (I should hedge a bit here. It is possible to build patterns that a FSM-like engine will do very poorly on, but where a backtracking engine may not degrade as much. The prototypical example is large repetitions, e.g., `(foo){100}`. Of course, the reverse is true as well, since FSMs don't have exponential worst case time. Also, my benchmarks aren't quite exhaustive. For example, they don't benchmark GNU grep/ripgrep's `-f` flag for searching many regexes.)

[1] - https://github.com/BurntSushi/ripgrep/blob/master/benchsuite...

[2] - https://github.com/petdance/ack3/issues/42

At some point I'll figure out a better alternative than this or pgrep which often misses processes.

Since this is an uncommon feature, I'd like to emphasize this. :-) In particular, ripgrep will automatically search UTF-16 encoded files via BOM sniffing. That means you can run ripgrep over a directory on Windows and be confident that it will correctly search both UTF-8 and UTF-16 encoded files automatically without having to think about it.

More generally, it supports all encodings found in the Encoding Standard[1]. However, only UTF-16 is automatically detected (where UTF-8 is the presumed default), so you'll need to explicitly specify `-E sjis` (for example) if you want to search Shift_JIS encoded files.

Also, I didn't really need to do much of anything to get this working. This is all thanks to @hsivonen's encoding_rs[2] crate, which is now (I think) in Firefox.

[1] - https://encoding.spec.whatwg.org/#concept-encoding-get

[2] - https://github.com/hsivonen/encoding_rs