Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

github.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#51

Post by gcr »

burntsushi wrote: post #48
shellac wrote: post #46

Earlier quoted context omitted.

The (standard) ripgrep regex engine has full unicode support. My reading of that is that it should handle such equivalences like matching the decomposed version.

It does not. Almost no regex engine does that. To add more color to this, the precise details of what "Unicode support" means are documented here: https://github.com/rust-lang/regex/blob/master/UNICODE.md In effect, all of UTS#18 Level 1 is covered with a couple caveats. This is already a far cry better than most regex engines, like PCRE2, which has limited support for properties and no way to do subtraction or inter…

For the purpose of searching, wouldn’t it be sufficient to do NFC normalization for text? Could hide that behind a command line flag even…

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#52

Post by burntsushi »

gcr wrote: post #51
burntsushi wrote: post #48

Earlier quoted context omitted.

It does not. Almost no regex engine does that. To add more color to this, the precise details of what "Unicode support" means are documented here: https://github.com/rust-lang/regex/blob/master/UNICODE.md In effect, all of UTS#18 Level 1 is covered with a couple caveats. This is already a far cry better than most regex engines, like PCRE2, which has limited support for properties and no way to do subtraction or inter…

For the purpose of searching, wouldn’t it be sufficient to do NFC normalization for text? Could hide that behind a command line flag even…

Can you say how that differs from what I suggested in my last paragraph? I legitimately can't tell if you're trying to suggest something different or not.

As UTS#18 2.1 says, it isn't sufficient to just normalize the text you're searching. It also means the user has to craft their regex appropriately. If you normalize to NFC but your regex uses NFD, oops. So it's probably best to expose a flag that lets you pick the normalization form.

And yes, it would have to be behind a CLI flag. Always doing normalization would likely make ripgrep slower than a naive grep written in Python. Yes. That bad. Yes. Really. And it might now become clear why a lot of tools don't do this.

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#53

Post by nanna »

medoc wrote: post #41
nullifidian wrote: post #15

It's somewhat similar to 'recoll' in its functionality, only with recoll you need to index everything before search. It even uses the same approach of using third-party software like poppler for extracting the contents.

By the way Recoll also has a utility named rclgrep which is an index-less search. It does everything that Recoll can do which can reasonably done without an index (e.g.: no proximity search, no stem expansion etc.). It will search all file types supported by Recoll, including embedded documents (email attachments, archive members, etc.). It is not built or distributed by default, because I think that building an inde…

Wow this is a gem of a comment. I use Recoll heavily, it's a real super power for an academic, but I had no idea about rclgrep. Thank you for all your work.

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#54

Post by nanna »

setopt wrote: post #21
nanna wrote: post #13

Lazy question but anyone integrated this with Emacs Dired, to transparently search all the files?

According to Reddit [1], you can use the existing rg.el package, and just point it to the rga binary instead of the rg binary, and it is supposed to just work. [1]: https://www.reddit.com/r/emacs/comments/1eghspj/comment/lg6q...

Huh, thanks yeah that does switch the binary to rga, but with rga you need to specify a wildcard operator for the path parameter in order to search PDFs, otherwise it only searches plaintext files, and I'm not sure how to make rg.el's RG function add that... must be a variable but not finding it.

  (use-package rg
    ;; ripgrep
    :ensure t
    :config
    (setq rg-executable (executable-find "rga")
     rg-buffer-name "rga"))

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#55

Post by justinmayer »

Integrating ripgrep-all with fzf makes for a powerful combination when you want to recursively search the contents of a given directory.

I have been using a shell function to do this, and it works wonderfully well: https://github.com/phiresky/ripgrep-all/wiki/fzf-Integration

The built-in rga-fzf command appeared in v0.10 and ostensibly obviates the need for the above shell function, but the built-in command produces errors for me on MacOS: https://github.com/phiresky/ripgrep-all/issues/240

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#56

Post by medoc »

nanna wrote: post #53
medoc wrote: post #41

Earlier quoted context omitted.

By the way Recoll also has a utility named rclgrep which is an index-less search. It does everything that Recoll can do which can reasonably done without an index (e.g.: no proximity search, no stem expansion etc.). It will search all file types supported by Recoll, including embedded documents (email attachments, archive members, etc.). It is not built or distributed by default, because I think that building an inde…

Wow this is a gem of a comment. I use Recoll heavily, it's a real super power for an academic, but I had no idea about rclgrep. Thank you for all your work.

What rclgrep does is run the recoll text extraction and do a grep-like operation on the extracted texts. If you want to give it a try, don't hesitate to contact me if you have any trouble building or using it, or think it could be improved. It's more or less a (working) prototype at the moment, but I'm willing to expand it if it looks useful. The "Usage" string is much better in the latest git source than in the tar, and it sorely needs a man page.

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#57

Post by shellac »

burntsushi wrote: post #48
shellac wrote: post #46

Earlier quoted context omitted.

The (standard) ripgrep regex engine has full unicode support. My reading of that is that it should handle such equivalences like matching the decomposed version.

It does not. Almost no regex engine does that. To add more color to this, the precise details of what "Unicode support" means are documented here: https://github.com/rust-lang/regex/blob/master/UNICODE.md In effect, all of UTS#18 Level 1 is covered with a couple caveats. This is already a far cry better than most regex engines, like PCRE2, which has limited support for properties and no way to do subtraction or inter…

Thanks very much for clarifying that. It did seem unlikely: I remember NSString (ask you parents...) supported this level of unicode equivalence, and it was quite a burden. Normalising does feel like the only tractable method here, and if you have an extraction pipeline anyway (in rga) maybe it's not so bad.

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#58

Post by burntsushi »

shellac wrote: post #57
burntsushi wrote: post #48

Earlier quoted context omitted.

It does not. Almost no regex engine does that. To add more color to this, the precise details of what "Unicode support" means are documented here: https://github.com/rust-lang/regex/blob/master/UNICODE.md In effect, all of UTS#18 Level 1 is covered with a couple caveats. This is already a far cry better than most regex engines, like PCRE2, which has limited support for properties and no way to do subtraction or inter…

Thanks very much for clarifying that. It did seem unlikely: I remember NSString (ask you parents...) supported this level of unicode equivalence, and it was quite a burden. Normalising does feel like the only tractable method here, and if you have an extraction pipeline anyway (in rga) maybe it's not so bad.

Yes, rga could support this in a more streamlined manner than rg, since rga has that extraction pipeline with caching. ripgrep just has a hook for an extraction pipeline.

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#59

Post by bomewish »

justinmayer wrote: post #55

Integrating ripgrep-all with fzf makes for a powerful combination when you want to recursively search the contents of a given directory. I have been using a shell function to do this, and it works wonderfully well: https://github.com/phiresky/ripgrep-all/wiki/fzf-Integration The built-in rga-fzf command appeared in v0.10 and ostensibly obviates the need for the above shell function, but the built-in command produces…

I also have a custom rga-fzf function, I think adapted from that:

rga-fzf () { local query=$1 local extension=$2 if [ -z "$extension" ] then RG_PREFIX="rga --files-with-matches --no-ignore" else RG_PREFIX="rga --files-with-matches --no-ignore --glob '*.$extension'" fi echo "RG Prefix: $RG_PREFIX" echo "Search Query: $query" FZF_DEFAULT_COMMAND="$RG_PREFIX '$query'" fzf --sort --preview="[[ ! -z {} ]] && rga --colors 'match:bg:yellow' --pretty --context 15 {q} {} | less -R" --multi --phony -q "$query" --color "hl:-1:underline,hl+:-1:underline:reverse" --bind "change:reload:$RG_PREFIX {q}" --preview-window="50%:wrap" --bind "enter:execute-silent(echo {} | xargs -n 1 open -g)" }

It allows one to continually open lots of files found through rga-fzf, so one can look at them in $EDITOR all at once. Useful sometimes.

Post reply on HN