Live data from Hacker News

Ripgrep – A new command line search tool

blog.burntsushi.net

111–120 of 219 posts

Re: Ripgrep – A new command line search tool

#111

Earlier quoted context omitted.

Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory m…

I notice that subsequent runs in the same (non changing) directory get different results. These runs are all within 20 seconds, what gives? $ rg each | md5sum 670b544e15f9430d9934334a11a87b7e - $ rg each | md5sum 4d13be6b4531ad52b1b476314fe98fb7 - rg each | md5sum 88e15dbb943665ea54482cb499741938 - rg each | md5sum eec6d6d5c9a592cec25aa8b0c19aae15 - rg each | md5sum ad74b78ef8f0d21450f8f87415555af0 - And: $ date Sat…

In the first case, it's searching in parallel, so I bet the order of results is different each time.

In the second case, rg each > foo2 found results in foo1 and put them in foo2. Then rg each > foo3 found results in foo1 and foo2, and put them in foo3. Etc. That's why the file size increases so quickly.

Re: Ripgrep – A new command line search tool

#112
post #111

Earlier quoted context omitted.

I notice that subsequent runs in the same (non changing) directory get different results. These runs are all within 20 seconds, what gives? $ rg each | md5sum 670b544e15f9430d9934334a11a87b7e - $ rg each | md5sum 4d13be6b4531ad52b1b476314fe98fb7 - rg each | md5sum 88e15dbb943665ea54482cb499741938 - rg each | md5sum eec6d6d5c9a592cec25aa8b0c19aae15 - rg each | md5sum ad74b78ef8f0d21450f8f87415555af0 - And: $ date Sat…

In the first case, it's searching in parallel, so I bet the order of results is different each time. In the second case, rg each > foo2 found results in foo1 and put them in foo2. Then rg each > foo3 found results in foo1 and foo2, and put them in foo3. Etc. That's why the file size increases so quickly.

>In the first case, it's searching in parallel, so I bet the order of results is different.

Aha. Thought that needed the -j flag (it says: default threads: 0 in the cli help).

Could it do anything to put them out in order of "depth" (and directory/file sorting order)?

>In the second case, rg each > foo2 found results in foo1 and put them in foo2. Then rg each > foo3 found results in foo1 and foo2, and put them in foo3. Etc. That's why the file size increases so quickly.

LOL, facepalm -- yes.

Re: Ripgrep – A new command line search tool

#113

Earlier quoted context omitted.

Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory m…

I notice that subsequent runs in the same (non changing) directory get different results. These runs are all within 20 seconds, what gives? $ rg each | md5sum 670b544e15f9430d9934334a11a87b7e - $ rg each | md5sum 4d13be6b4531ad52b1b476314fe98fb7 - rg each | md5sum 88e15dbb943665ea54482cb499741938 - rg each | md5sum eec6d6d5c9a592cec25aa8b0c19aae15 - rg each | md5sum ad74b78ef8f0d21450f8f87415555af0 - And: $ date Sat…

This can happen because rg searches files in parallel, so the order in which it finishes the files can be nondeterministic. If you run with -j1 (single-threaded) then it is deterministic.

To get deterministic output in multi-threaded mode, rg could wait and buffer the output until it can print it in sorted order. This might increase memory usage, and possibly time, though I think the increase would be minor.

Re: Ripgrep – A new command line search tool

#114
post #111

Earlier quoted context omitted.

In the first case, it's searching in parallel, so I bet the order of results is different each time. In the second case, rg each > foo2 found results in foo1 and put them in foo2. Then rg each > foo3 found results in foo1 and foo2, and put them in foo3. Etc. That's why the file size increases so quickly.

> In the first case, it's searching in parallel, so I bet the order of results is different. Aha. Thought that needed the -j flag (it says: default threads: 0 in the cli help). Could it do anything to put them out in order of "depth" (and directory/file sorting order)? > In the second case, rg each > foo2 found results in foo1 and put them in foo2. Then rg each > foo3 found results in foo1 and foo2, and put them in f…

Forcing it to use one worker (-j1, I think) should give it deterministic output.

Re: Ripgrep – A new command line search tool

#115
post #103

Earlier quoted context omitted.

Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory m…

In terms of core features, ripgrep is totally there. It searches fast . It ignores files pretty accurately. It outputs results in a pleasant and useful format. If a new user tries rg, they'll be very happy. My warning about the feature differences was meant to temper ag users' expectations. There are lots of little things that ag users are accustomed to that are either different or missing in ripgrep. Off the top of…

I'd say that .ignore is too generic a name. What is to be ignored by what?

But I like the idea of standardizing this. Perhaps the cache directory tagging standard gives some inspiration.

http://www.brynosaurus.com/cachedir/spec.html

Re: Ripgrep – A new command line search tool

#116
post #63

I'm the author of ag. That was a really good comparison of the different code searching tools. The author did a great job of showing how each tool misbehaved or performed poorly in certain circumstances. He's also totally right about defaults mattering. It looks like ripgrep gets most of its speedup on ag by: 1. Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are…

> Only supporting DFA-able Rust regexes. I'd love to use a lighter-weight regex library in ag, but users are accustomed to full PCRE support Would it be possible to detect when an expression requires PCRE-specific features and use a different engine when possible?

It's possible, but it's certainly not easy. Here are some complications:

1. The DFA-regex engine's syntax must be a subset of PCRE's syntax. If it's not, then users will be very confused when regex features work fine in isolation, but cause errors when combined in the same query.

2. The DFA-regex's behavior must be the same as PCRE. If whitespace matching or unicode support is even slightly different, it will frustrate users.

3. Adding another dependency means yet another way in which compilation can fail or incompatibilities can arise.

Considering the marginal usefulness of backtracking and captures, I'd prefer to keep ag as simple as possible and ditch them.

Re: Ripgrep – A new command line search tool

#117
post #103

Earlier quoted context omitted.

Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory m…

In terms of core features, ripgrep is totally there. It searches fast . It ignores files pretty accurately. It outputs results in a pleasant and useful format. If a new user tries rg, they'll be very happy. My warning about the feature differences was meant to temper ag users' expectations. There are lots of little things that ag users are accustomed to that are either different or missing in ripgrep. Off the top of…

Could I suggest .ignorerc?

I would immediately intuit that such a file is a (r)untime (c)onfiguration for ignoring something.

.ignore isn't... bad, it just looks like something I can safely delete, like a file styled `such-and-such~`

Re: Ripgrep – A new command line search tool

#118

Earlier quoted context omitted.

I'd like to get this working too, since I know a lot of folks are happy with it for ag and ack. rg does have a --vimgrep option that should make it as easy as ag to use, but I don't think there is a proper integration just yet.

I like ripgrep quite a bit from trying it today, and am hoping to find time to work on a Vim plugin this weekend. No promises, but I'll share as soon as I have something usable.

That's fantastic! Please don't hesitate to file an issue if you run into problems.

Re: Ripgrep – A new command line search tool

#119
post #103

Earlier quoted context omitted.

Thanks for the response! Some notes: 1. In my benchmarks, I do control for line numbers by either explicitly making it a variable (i.e., when you see `(lines)`) or by making all tools count lines to make the comparison fair. For the most part, this only tends to matter in the single-file benchmarks. 2. For memory maps, you might get very different results depending on your environment. For example, I enabled memory m…

In terms of core features, ripgrep is totally there. It searches fast . It ignores files pretty accurately. It outputs results in a pleasant and useful format. If a new user tries rg, they'll be very happy. My warning about the feature differences was meant to temper ag users' expectations. There are lots of little things that ag users are accustomed to that are either different or missing in ripgrep. Off the top of…

May be .searchignore?

Re: Ripgrep – A new command line search tool

#120
post #106

... $ rg -uu foobar # similar to `grep -r` $ rg -uuu foobar # similar to `grep -a -r` I knew it. The name is absolutely ironic. I cannot just drop-it-in and make all my scripts and whatever scripts I download work immediately faster (nor is it compatible with my shell typing reflexes). New, shiny, fast tool, doomed from birth.

It fundamentally can't be interface compatible, sorry. I think I was pretty clear about this in the blog. :-)
Post reply on HN