Live data from Hacker News

Ripgrep – A new command line search tool

blog.burntsushi.net

91–100 of 219 posts

Re: Ripgrep – A new command line search tool

#91

Very nice. Not only fast, but feels modern. Tried it out on a 3.5GB JSON file: # rg rg erzg4 k.json > /dev/null 1.80s user 2.54s system 53% cpu 8.053 total # rg with 4 threads rg -j4 erzg4 k.json > /dev/null 1.76s user 1.29s system 99% cpu 3.059 total # OS X grep grep erzg4 k.json > /dev/null 60.62s user 0.96s system 99% cpu 1:01.75 total # GNU Grep ggrep erzg4 k.json > /dev/null 1.96s user 1.43s system 88% cpu 2.691…

My guess is that since you ran `rg` first, the file wasn't in memory, and you ended up benchmarking disk IO. (Notice the sys time decrease from your first run to the second run.) Subsequent commands then run faster with the file already in memory. This is one of many reasons why assembling the benchmarks in my blog post was so difficult. For example, on every command I benchmarked, I ran them 3 times for "warmup" and…

I ran each test four times and picked the best result — not my first rodeo — but for the first result I picked the wrong time from my output, which obviously didn't make use of the cache. Here's it again, complete result, added --no-mmap:

    # ggrep
    zerogravitas$ for x in {1..4}; do (time ggrep erzg4 k.json >/dev/null); done
    ggrep erzg4 k.json > /dev/null  1.96s user 0.67s system 99% cpu 2.641 total
    ggrep erzg4 k.json > /dev/null  1.95s user 0.68s system 99% cpu 2.660 total
    ggrep erzg4 k.json > /dev/null  2.00s user 0.66s system 99% cpu 2.672 total
    ggrep erzg4 k.json > /dev/null  1.96s user 0.67s system 98% cpu 2.662 total

    # rg
    zerogravitas$ for x in {1..4}; do (time rg erzg4 k.json >/dev/null); done
    rg erzg4 k.json > /dev/null  1.76s user 1.40s system 99% cpu 3.180 total
    rg erzg4 k.json > /dev/null  1.77s user 1.31s system 99% cpu 3.088 total
    rg erzg4 k.json > /dev/null  1.74s user 1.36s system 99% cpu 3.128 total
    rg erzg4 k.json > /dev/null  1.76s user 1.41s system 97% cpu 3.265 total

    # rg --no-mmap
    zerogravitas$ for x in {1..4}; do (time rg erzg4 k.json --no-mmap >/dev/null); done
    rg erzg4 k.json --no-mmap > /dev/null  0.98s user 0.75s system 99% cpu 1.743 total
    rg erzg4 k.json --no-mmap > /dev/null  0.99s user 0.75s system 99% cpu 1.740 total
    rg erzg4 k.json --no-mmap > /dev/null  1.01s user 0.76s system 99% cpu 1.772 total
    rg erzg4 k.json --no-mmap > /dev/null  0.99s user 0.75s system 99% cpu 1.754 total

    # rg -j4
    zerogravitas$ for x in {1..4}; do (time rg erzg4 k.json -j4 >/dev/null); done
    rg erzg4 k.json -j4 > /dev/null  1.75s user 1.35s system 98% cpu 3.134 total
    rg erzg4 k.json -j4 > /dev/null  1.75s user 1.44s system 98% cpu 3.224 total
    rg erzg4 k.json -j4 > /dev/null  1.80s user 1.38s system 99% cpu 3.204 total
    rg erzg4 k.json -j4 > /dev/null  1.80s user 1.35s system 99% cpu 3.164 total

    # rg -j4 --no-mmap
    zerogravitas$ for x in {1..4}; do (time rg erzg4 k.json -j4 --no-mmap >/dev/null); done
    rg erzg4 k.json -j4 --no-mmap > /dev/null  0.98s user 0.75s system 99% cpu 1.740 total
    rg erzg4 k.json -j4 --no-mmap > /dev/null  0.97s user 0.74s system 99% cpu 1.721 total
    rg erzg4 k.json -j4 --no-mmap > /dev/null  0.99s user 0.75s system 99% cpu 1.752 total
    rg erzg4 k.json -j4 --no-mmap > /dev/null  0.98s user 0.76s system 99% cpu 1.748 total
Sounds like "alias rg=rg --no-mmap" is a good idea on a Mac.

Re: Ripgrep – A new command line search tool

#92
post #20

Earlier quoted context omitted.

Honestly, for a certain kind of developer—I haven't fingerpointed exactly which kind—writing Rust code is dangerously addictive. There's some combination of aggressive type checking, tooling, and expressiveness that just hits a sweet spot in my brain. Rust fills roughly the same niche in my brain as old-school C++, except it also lets me do some rudimentary functional stuff, there are virtually no footguns, and the C…

Well in the interests of making contraversal statements, that's now two Rust programs I plan on using day-to-day, and zero for Haskell, Scala, and I'm sure a few other languages I'm not remembering.

Pandoc

Xmonad

Darcs

Git-annex

Although, it would be very hard to write SIMD performant code in Haskell, even its string libraries don't use any optimization tricks.

Re: Ripgrep – A new command line search tool

#93

Anyone have any suggestions regarding how to best use Ripgrep within Vim? Specifically, how best to use it to recursively search the current directory (or specified directory) and have the results appear in a quickfix window that allows for easily opening the file(s) that contain the searched term.

I'd like to get this working too, since I know a lot of folks are happy with it for ag and ack. rg does have a --vimgrep option that should make it as easy as ag to use, but I don't think there is a proper integration just yet.

Re: Ripgrep – A new command line search tool

#94

Earlier quoted context omitted.

My guess is that since you ran `rg` first, the file wasn't in memory, and you ended up benchmarking disk IO. (Notice the sys time decrease from your first run to the second run.) Subsequent commands then run faster with the file already in memory. This is one of many reasons why assembling the benchmarks in my blog post was so difficult. For example, on every command I benchmarked, I ran them 3 times for "warmup" and…

I ran each test four times and picked the best result — not my first rodeo — but for the first result I picked the wrong time from my output, which obviously didn't make use of the cache. Here's it again, complete result, added --no-mmap: # ggrep zerogravitas$ for x in {1..4}; do (time ggrep erzg4 k.json >/dev/null); done ggrep erzg4 k.json > /dev/null 1.96s user 0.67s system 99% cpu 2.641 total ggrep erzg4 k.json >…

Wow. Those are awesome results, thank you.

> Sounds like "alias rg=rg --no-mmap" is a good idea on a Mac.

I will fix that in ripgrep proper by making --no-mmap the default on Mac. :-) It should be an easy one to knock off: https://github.com/BurntSushi/ripgrep/issues/36

> not my first rodeo

Right, sorry about that. :-) Just had to cover all my bases!

(Also, `-j` on a single file won't do anything, and ripgrep should try to use multiple threads by default when searching multiple files.)

Re: Ripgrep – A new command line search tool

#95
post #63

I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or performed poorly in certain circumstances. He's also totally right about defaults mattering. It looks like ripgrep gets most of its speedup on ag by: 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are…

For anyone else interested in the memory map issue, here's some more data: https://news.ycombinator.com/item?id=12567326

Re: Ripgrep – A new command line search tool

#96
On a somewhat related note.

There does not appear be a popular indexed full-text search tool in existence.

Think cross-platform version of Spotlight's mdfind. Could there be something fundamental that makes this approach unsuitable for code search?

Alternatively, something like locate, but realtime and fulltext, instead of filename only.

Re: Ripgrep – A new command line search tool

#97

Earlier quoted context omitted.

I ran each test four times and picked the best result — not my first rodeo — but for the first result I picked the wrong time from my output, which obviously didn't make use of the cache. Here's it again, complete result, added --no-mmap: # ggrep zerogravitas$ for x in {1..4}; do (time ggrep erzg4 k.json >/dev/null); done ggrep erzg4 k.json > /dev/null 1.96s user 0.67s system 99% cpu 2.641 total ggrep erzg4 k.json >…

Wow. Those are awesome results, thank you. > Sounds like "alias rg=rg --no-mmap" is a good idea on a Mac. I will fix that in ripgrep proper by making --no-mmap the default on Mac. :-) It should be an easy one to knock off: https://github.com/BurntSushi/ripgrep/issues/36 > not my first rodeo Right, sorry about that. :-) Just had to cover all my bases! (Also, `-j` on a single file won't do anything, and ripgrep should…

I suspected that -j wouldn't do anything on a single file. For three large (6.5GB in total) files I'm getting good performance, about 1.6x of what GNU Grep does, best case.

Re: Ripgrep – A new command line search tool

#98
post #96

On a somewhat related note. There does not appear be a popular indexed full-text search tool in existence. Think cross-platform version of Spotlight's mdfind . Could there be something fundamental that makes this approach unsuitable for code search? Alternatively, something like locate , but realtime and fulltext, instead of filename only.

AFAIK mdfind heavily depends on file system events in macOS, it would be painful or impossible to implement such system in cross platform way with support for file systems like FAT etc.
Post reply on HN