Live data from Hacker News

Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

github.com

41–50 of 59 posts

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#41

It's somewhat similar to 'recoll' in its functionality, only with recoll you need to index everything before search. It even uses the same approach of using third-party software like poppler for extracting the contents.

By the way Recoll also has a utility named rclgrep which is an index-less search. It does everything that Recoll can do which can reasonably done without an index (e.g.: no proximity search, no stem expansion etc.). It will search all file types supported by Recoll, including embedded documents (email attachments, archive members, etc.). It is not built or distributed by default, because I think that building an index is a better approach, but it's in the source tar distribution and can be built with -Drclgrep=true. Disclosure: I am the Recoll developper.

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#42
post #10

Earlier quoted context omitted.

Also note that it's not necessarily safe to read these documents even if you don't intend on executing embedded code. For example, reading from pdfs uses poppler, which has had a few CVEs that could result in arbitrary code execution, mostly around image decoding. https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=poppler (No shade to poppler intended, just the first tool on the list I looked at.)

Couldn't or shouldn't each parser be run in a container with systemd-nspawn or LXC or another container runtime? (Even if all it's doing is reading a file format into process space as NX data not code as the current user)

NX bit: https://en.wikipedia.org/wiki/NX_bit

Executable-space protection > Limitations mentions JITs and ROP: https://en.wikipedia.org/wiki/Executable-space_protection

mprotect(), VirtualAlloc[Ex] and VirtualProtect[Ex],

"NX bit: does it protect the stack?" https://security.stackexchange.com/questions/47807/nx-bit-do...

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#43

Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…

Seems to work well when it's searching the PDF text layer as ligatures are a font rendering effect. You're right — ligatures are not as common in modern books.

Might be iffier in OCR mode: it seems to use Tesseract, which is known to have issues recognising ligatured text.

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#44

It's somewhat similar to 'recoll' in its functionality, only with recoll you need to index everything before search. It even uses the same approach of using third-party software like poppler for extracting the contents.

I think an index of all documents (including the contained text etc) should be a standardized component / API of every modern OS. Windows has had one since Vista (no idea about the API though), Spotlight has been a part of OS X for two decades, and there are various solutions for Linux & friends; however as far as I can tell there's no cross-platform wrapper that would make any or all of these easy to integrate with e.g. your IDE. That would be cool to have.

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#45

Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…

> I notice that the word "Office" in the title of this article is not rendered with a ligature by Chrome

Chrome mobile on Android does render Office with what looks like at least an fi ligature for me (it should use an ffi one but still).

Maybe it depends on the font?

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#46

Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…

The (standard) ripgrep regex engine has full unicode support. My reading of that is that it should handle such equivalences like matching the decomposed version.

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#48
post #46

Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…

The (standard) ripgrep regex engine has full unicode support. My reading of that is that it should handle such equivalences like matching the decomposed version.

It does not. Almost no regex engine does that.

To add more color to this, the precise details of what "Unicode support" means are documented here: https://github.com/rust-lang/regex/blob/master/UNICODE.md

In effect, all of UTS#18 Level 1 is covered with a couple caveats. This is already a far cry better than most regex engines, like PCRE2, which has limited support for properties and no way to do subtraction or intersection of character classes. Other regex engines, like Javascript, are catching up. While UTS#18 Level 1 make ripgrep's Unicode support better than most, it does not make it the best. The third party Python `regex` library, for example, has very good support, although it is not especially fast[1].

Short of building UTS#18 2.1[2] support into the regex engine (unlikely to ever happen), it's likely ripgrep could offer some sort of escape hatch. Perhaps, for example, an option to normalize all text searched to whatever form you want (nfc, nfd, nfkc or nfkd). The onus would still be on you to write the corresponding regex pattern though. You can technically do this today with ripgrep's `--pre` flag, but having something built-in might be nice. Indeed, if you read UTS#18 2.1, you'll note that it is self-aware about how difficult matching canonical equivalents is, and essentially suggests this exact work-around instead. The problem is that it would need to be opt-in and the user would need to be aware of the problem in the first place. That's... a stretch, but probably better than nothing.

[1]: https://github.com/BurntSushi/rebar?tab=readme-ov-file#summa...

[2]: https://unicode.org/reports/tr18/#Canonical_Equivalents

Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.

#49

Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…

> I don't have any sense of how common ligature usage is anymore

It's much more common in PDFs than it is on the web, at least when the underlying plaintext is concerned.

Post reply on HN