Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ripgrep – A new command line search tool

blog.burntsushi.net

Post reply on HN
Print view View original

81–90 of 219 posts

Re: Ripgrep – A new command line search tool

#81

Post by burntsushi »

kozikow wrote: post #73

1. Ag have nice editor integration. I would miss emacs helm-projectile-ag 2. Pcre is good regexp flavor to master. It is have good balance of speed, power and popularity. In addition to Ag, there are accessible libraries in many languages, including python. I think it would be good if everyone settled on Pcre, rather than each language thinking they will do regexps better.

PCRE suffers from worst case exponential behavior, so it's not suitable for all tasks.

For the most part, the syntax supported by ripgrep is a strict subset of the syntax supported by PCRE.

But yes, I can agree that supporting PCRE can be considered an advantage if you use advanced features heavily (backreferences and lookaround come to mind).

Re: Ripgrep – A new command line search tool

#82

Post by sitkack »

CJefferson wrote: post #24
ekidd wrote: post #20

Earlier quoted context omitted.

Honestly, for a certain kind of developer—I haven't fingerpointed exactly which kind—writing Rust code is dangerously addictive. There's some combination of aggressive type checking, tooling, and expressiveness that just hits a sweet spot in my brain. Rust fills roughly the same niche in my brain as old-school C++, except it also lets me do some rudimentary functional stuff, there are virtually no footguns, and the C…

Well in the interests of making contraversal statements, that's now two Rust programs I plan on using day-to-day, and zero for Haskell, Scala, and I'm sure a few other languages I'm not remembering.

I need an R5RS+ Scheme that is tightly integrated with Rust. High-level and fast enough, with the ability to drop down into Rust for speed.

Re: Ripgrep – A new command line search tool

#83

Post by burntsushi »

krylon wrote: post #42

When I use grep (which is fairly regularly), the bottleneck is nearly always the disk or the network (in case of NFS/SMB volumes). Just out of curiosity, what kind of use case makes grep and prospective replacements scream? The most "hardcore" I got with grep was digging through a few gigabytes of ShamePoint logs looking for those correlation IDs, and that apparently was completely I/O-bound, the CPUs on that machine…

> Just out of curiosity, what kind of use case makes grep and prospective replacements scream?

Unicode? Check out the subtitle benchmarks in the blog post. In the best case, grep is a little slower. In the worst case, grep is orders of magnitude slower.

ripgrep achieves speed by building UTF-8 decoding straight into its DFA regex engine (well, strictly speaking, this is Rust's regex engine, not ripgrep).

The other case where grep users might scream is when you're searching large code repositories. A `grep -r` might catch a large binary file, or search your `.git` or whatever. Both `ag` and `rg` will look at your `.gitignore` so that the results you see have higher relevance. (Of course, this is just a default, you can always "search everything" with ripgrep too!)

Re: Ripgrep – A new command line search tool

#84

Post by burntsushi »

nolemurs wrote: post #44
krylon wrote: post #42

When I use grep (which is fairly regularly), the bottleneck is nearly always the disk or the network (in case of NFS/SMB volumes). Just out of curiosity, what kind of use case makes grep and prospective replacements scream? The most "hardcore" I got with grep was digging through a few gigabytes of ShamePoint logs looking for those correlation IDs, and that apparently was completely I/O-bound, the CPUs on that machine…

`grep` itself is admirably fast on single files, and if you're just searching a small number of files, the bottleneck will be IO, and `grep` will be about as fast as anything. But if you want to use `grep` recursively on a selected subset of millions of files it suffers a bit. `grep` assumes that if you want to collect some subset of files to search, you'll do that with some other utility, and pass the list of files…

> and `grep` will be about as fast as anything

I'd encourage you to check out the benchmarks in my blog, especially the subtitle benchmarks, because this isn't actually true in the case of ripgrep. :-)

Re: Ripgrep – A new command line search tool

#85

Post by sitkack »

chalana wrote: post #27

I'm never sure whether or not I should adopt these fancy new command line tools that come out. I get them on my muscle memory and then all of a sudden I ssh into a machine that doesn't have any of these and I'm screwed...

Just make your ssh script install all the binaries where ever you go! ssh-caravan

Re: Ripgrep – A new command line search tool

#86

Post by sidlls »

burntsushi wrote: post #79
sidlls wrote: post #53

Earlier quoted context omitted.

Is there anything about ripgrep that makes its performance uniquely fast? That is, is it not possible to achieve the same speed and correctness of results using, say, C or C++? If neither condition holds, then the use of Rust is almost purely incidental. It isn't even interesting, from that perspective. Far more interesting would be a measure of how safe the Rust code is versus equivalent-results implementations in o…

I can't think of any particular piece that requires Rust specifically for performance, no. The real beauty of Rust is that I was able to achieve this performance in the first place, all with very very little use of unsafe (even in the regex engine itself). But of course, as we all know, performance isn't everything. Even ripgrep itself accepts a performance penalty in its default mode of operation, in order to improv…

Yes, that was my main point, re: safety. I've been implementing a BLAS with Rust (pure Rust--not FFI wrappers around C-wrappers around Fortan or other bindings) and it has been interesting to see where the C-programmer in me gets tripped up and how the performance compares.

Re: Ripgrep – A new command line search tool

#87

Post by burntsushi »

lobster_johnson wrote: post #62

Very nice. Not only fast, but feels modern. Tried it out on a 3.5GB JSON file: # rg rg erzg4 k.json > /dev/null 1.80s user 2.54s system 53% cpu 8.053 total # rg with 4 threads rg -j4 erzg4 k.json > /dev/null 1.76s user 1.29s system 99% cpu 3.059 total # OS X grep grep erzg4 k.json > /dev/null 60.62s user 0.96s system 99% cpu 1:01.75 total # GNU Grep ggrep erzg4 k.json > /dev/null 1.96s user 1.43s system 88% cpu 2.691…

My guess is that since you ran `rg` first, the file wasn't in memory, and you ended up benchmarking disk IO. (Notice the sys time decrease from your first run to the second run.) Subsequent commands then run faster with the file already in memory.

This is one of many reasons why assembling the benchmarks in my blog post was so difficult. For example, on every command I benchmarked, I ran them 3 times for "warmup" and didn't record any measurements. I then ran them another 10 times in which I recorded them. You can see the raw output here: https://github.com/BurntSushi/ripgrep/blob/master/benchsuite...

In any case, on my underpowered Mac, here are some results on a 1.2 GB file (notice how much the time fluctuates until its fully in cache):

    mac:~ andrew$ ggrep --version
    ggrep (GNU grep) 2.25                                                                                                                                                                                                
    Packaged by Homebrew
    mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null   
    
    real    0m5.447s
    user    0m0.600s
    sys     0m0.350s
    mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null
    
    real    0m1.247s
    user    0m0.549s
    sys     0m0.264s
    mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null
    
    real    0m0.803s
    user    0m0.542s
    sys     0m0.259s
    mac:~ andrew$ time ggrep 'Bruce Springsteen' foo.jsonl > /dev/null
    
    real    0m0.805s
    user    0m0.544s
    sys     0m0.260s
And now for rg:

    mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null
    
    real    0m1.062s
    user    0m0.339s
    sys     0m0.333s
    mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null
    
    real    0m0.640s
    user    0m0.337s
    sys     0m0.302s
    mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl > /dev/null
    
    real    0m0.637s
    user    0m0.336s
    sys     0m0.300s
Oh! And check this out, on a Mac, not using a memory map for single files is faster. My goodness---memory map performance is all over the place.

    mac:~ andrew$ time rg 'Bruce Springsteen' foo.jsonl --no-mmap > /dev/null
    
    real    0m0.445s
    user    0m0.170s
    sys     0m0.274s
If I do this on my Linux machine on the same file, I get timings of 0.275s for rg, 0.398s for rg with no memory maps (opposite direction for Mac) and 0.708s for GNU grep (v 2.25).

Benchmarks are fun, eh?

Re: Ripgrep – A new command line search tool

#88

Post by FreeFull »

awesomebob wrote: post #59
steveklabnik wrote: post #40

Earlier quoted context omitted.

Negative time, actually: https://github.com/kintaro/wtftw (there's an i3 inspired one as well, I forget the link)

Nice! I think this is the other one you were thinking of: https://github.com/tsurai/xr3wm

There is a different one based on wlc (So it's a wayland compositor, rather than just an X11 window manager). The one you linked doesn't seem to have been updated since 2015

Edit: Here it is: https://github.com/Immington-Industries/way-cooler

Re: Ripgrep – A new command line search tool

#89

Post by sidlls »

theseoafs wrote: post #57
sidlls wrote: post #53

Earlier quoted context omitted.

Is there anything about ripgrep that makes its performance uniquely fast? That is, is it not possible to achieve the same speed and correctness of results using, say, C or C++? If neither condition holds, then the use of Rust is almost purely incidental. It isn't even interesting, from that perspective. Far more interesting would be a measure of how safe the Rust code is versus equivalent-results implementations in o…

It seems to be faster than grep primarily by having fewer features (it doesn't even attempt to support all of POSIX regular expressions, for one).

I don't know about that (as in I haven't looked so I literally don't know). First of all, provided the tools produce identical results for similarly-complex regex patterns I'm not sure having to support all POSIX regexes is necessary. Second of all my point was really that the performance achievement is not nearly as interesting as the level of rust-guaranteed safety found as compared with similar implementations in other ("unsafe") languages.

Re: Ripgrep – A new command line search tool

#90

Post by justinmayer »

Anyone have any suggestions regarding how to best use Ripgrep within Vim? Specifically, how best to use it to recursively search the current directory (or specified directory) and have the results appear in a quickfix window that allows for easily opening the file(s) that contain the searched term.
Post reply on HN