Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ripgrep – A new command line search tool

blog.burntsushi.net

Post reply on HN
Print view View original

101–110 of 219 posts

Re: Ripgrep – A new command line search tool

#101

Post by cgag »

pmontra wrote: post #55
steveklabnik wrote: post #50

Earlier quoted context omitted.

If you don't want to compile it yourself, the blog post has links to binaries. Since they're entirely self-contained, you don't need a full .deb to compile them; just delete the binary when you want to get rid of it.

I didn't notice that. Thanks. It works. The binary is 11 MB, or 1.5 MB stripped. ag is 69176 bytes. Shared libraries were a good invention :-) Sooner or later rg will be packaged properly too.

I'd much rather have a self contained binary than 11 more MB of free space.

Re: Ripgrep – A new command line search tool

#102

Post by wamatt »

diimdeep wrote: post #98
wamatt wrote: post #96

On a somewhat related note. There does not appear be a popular indexed full-text search tool in existence. Think cross-platform version of Spotlight's mdfind . Could there be something fundamental that makes this approach unsuitable for code search? Alternatively, something like locate , but realtime and fulltext, instead of filename only.

AFAIK mdfind heavily depends on file system events in macOS, it would be painful or impossible to implement such system in cross platform way with support for file systems like FAT etc.

sure, FAT. OTOH some people think it's not a totally intractable problem

A cross-platform file change monitor with multiple backends: Apple OS X File System Events, BSD kqueue, Solaris/Illumos File Events Notification, Linux inotify, Microsoft Windows and a stat()-based backend.

http://emcrisostomo.github.io/fswatch/

https://github.com/emcrisostomo/fswatch

Re: Ripgrep – A new command line search tool

#103

Post by ggreer »

burntsushi wrote: post #66
ggreer wrote: post #63

I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or performed poorly in certain circumstances. He's also totally right about defaults mattering. It looks like ripgrep gets most of its speedup on ag by: 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are…

Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory m…

In terms of core features, ripgrep is totally there. It searches fast. It ignores files pretty accurately. It outputs results in a pleasant and useful format. If a new user tries rg, they'll be very happy.

My warning about the feature differences was meant to temper ag users' expectations. There are lots of little things that ag users are accustomed to that are either different or missing in ripgrep. Off the top of my head: Ag reads the user's global gitignore. (This is harder than most people think.) It detects stdout redirects such as "ag blah > output.txt" and ignores output.txt. It can search gz and xz files. It defaults to smart-case searching. It can limit a search to one hardware device (--one-device), avoiding slow reads on network mounts. And as a commenter already pointed out, it supports the --pager option. Taken together, all those small differences are likely to cause an average ag user some grief. I wanted to manage expectations so that users wouldn't create annoying "issues" (really, feature requests) on your GitHub repo. Sorry if that came off the wrong way.

On a completely unrelated note: I see ripgrep supports .rgignore files, similar to how ag supports .agignore. It'd be nice if we could combine forces and choose a single filename for this purpose. That way when the next search tool comes along, it can use the same thing instead of .zzignore or whatever. It would also make it easier for users to switch between our tools. I'd suggest a generic name like ".ignore" or ".ignores", but I'm sure some tool creates such files or directories already.

Edit: Actually, it looks like .ignore can work. The only examples I've found of .ignore files are actual files containing ignore patterns.

Re: Ripgrep – A new command line search tool

#104

Post by pcwalton »

kozikow wrote: post #73

1. Ag have nice editor integration. I would miss emacs helm-projectile-ag 2. Pcre is good regexp flavor to master. It is have good balance of speed, power and popularity. In addition to Ag, there are accessible libraries in many languages, including python. I think it would be good if everyone settled on Pcre, rather than each language thinking they will do regexps better.

> I think it would be good if everyone settled on Pcre, rather than each language thinking they will do regexps better.

On the contrary, I think we can do better than a big pile of C code: https://www.cvedetails.com/vulnerability-list/vendor_id-3265...

Re: Ripgrep – A new command line search tool

#105

Post by mintplant »

ggreer wrote: post #63

I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or performed poorly in certain circumstances. He's also totally right about defaults mattering. It looks like ripgrep gets most of its speedup on ag by: 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are…

> Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support

Would it be possible to detect when an expression requires PCRE-specific features and use a different engine when possible?

Re: Ripgrep – A new command line search tool

#106

Post by wruza » 

    ...
    $ rg -uu foobar  # similar to `grep -r`
    $ rg -uuu foobar  # similar to `grep -a -r`
I knew it. The name is absolutely ironic. I cannot just drop-it-in and make all my scripts and whatever scripts I download work immediately faster (nor is it compatible with my shell typing reflexes). New, shiny, fast tool, doomed from birth.

Re: Ripgrep – A new command line search tool

#107

Post by brandmeyer »

karmakaze wrote: post #61
bodyfour wrote: post #30

It would be interesting to benchmark how much mmap hurts when operating in a non-parallel mode. I think a lot of the residual love for mmap is because it actually did give decent results back when single core machines were the norm. However, once your program becomes multithreaded it imposes a lot of hidden synchronization costs, especially on munmap(). The fastest option might well be to use mmap sometimes but have…

I recently questioned if/ why parallel mmap might be slower without a satisfactory conclusion. One specific thing I couldn't answer is if reusing the same filesystem buffer and program memory addresses has a less negative effect than reading a wide range of mapped memory addresses.

Since all processors must share the mapping,

- The initial mapping of each file in any thread must halt all of the threads which are otherwise active.

- Every page fault in any mapping must also halt all of the threads.

Worse, since the page tables are getting munged, some or all of the TLB cache is getting flushed every time, again, on every processor.

I'm not sure of the details, but this hypothesis should be directly testable. IIRC, there are some hardware performance counters for time spent waiting on TLB lookups.

Addendum: One other possibility is that the mere act of extending the working set size (of the address space) is blowing the TLB cache.

Re: Ripgrep – A new command line search tool

#108

Post by coldtea »

burntsushi wrote: post #66
ggreer wrote: post #63

I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or performed poorly in certain circumstances. He's also totally right about defaults mattering. It looks like ripgrep gets most of its speedup on ag by: 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are…

Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory m…

I notice that subsequent runs in the same (non changing) directory get different results. These runs are all within 20 seconds, what gives?

  $ rg each | md5sum
  670b544e15f9430d9934334a11a87b7e  -
  $ rg each | md5sum
  4d13be6b4531ad52b1b476314fe98fb7  -
  rg each | md5sum
  88e15dbb943665ea54482cb499741938  -
  rg each | md5sum
  eec6d6d5c9a592cec25aa8b0c19aae15  -
  rg each | md5sum
  ad74b78ef8f0d21450f8f87415555af0  -
And:

  $ date
  Sat Sep 24 01:42:27 EEST 2016
  $ rg each > foo1
  $ rg each > foo2
  $ rg each > foo3
  $ rg each > foo4
  $ ls -la foo*
  -rw-r--r--  1 coldtea  staff  1429646 Sep 24 01:42 foo1
  -rw-r--r--  1 coldtea  staff  2250868 Sep 24 01:42 foo2
  -rw-r--r--  1 coldtea  staff  4536031 Sep 24 01:42 foo3
  -rw-r--r--  1 coldtea  staff  9140652 Sep 24 01:42 foo4
  $ date
  Sat Sep 24 01:42:44 EEST 2016
OS X 10.12, installed with brew.

Re: Ripgrep – A new command line search tool

#109

Post by nkantar »

burntsushi wrote: post #93
justinmayer wrote: post #90

Anyone have any suggestions regarding how to best use Ripgrep within Vim? Specifically, how best to use it to recursively search the current directory (or specified directory) and have the results appear in a quickfix window that allows for easily opening the file(s) that contain the searched term.

I'd like to get this working too, since I know a lot of folks are happy with it for ag and ack. rg does have a --vimgrep option that should make it as easy as ag to use, but I don't think there is a proper integration just yet.

I like ripgrep quite a bit from trying it today, and am hoping to find time to work on a Vim plugin this weekend. No promises, but I'll share as soon as I have something usable.

Re: Ripgrep – A new command line search tool

#110

Post by eel »

justinmayer wrote: post #90

Anyone have any suggestions regarding how to best use Ripgrep within Vim? Specifically, how best to use it to recursively search the current directory (or specified directory) and have the results appear in a quickfix window that allows for easily opening the file(s) that contain the searched term.

This is what I set in my .vimrc today, and it seems to be working pretty well:

  if executable('rg')
    set grepprg=rg\ --no-heading\ --vimgrep
    set grepformat=%f:%l:%c:%m
  endif
Then you can just :grep yourquery (or :grep yourquery path), and the results will show in a quickfix window.
Post reply on HN