It's somewhat similar to 'recoll' in its functionality, only with recoll you need to index everything before search. It even uses the same approach of using third-party software like poppler for extracting the contents.
Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.
41–50 of 59 posts
Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.
#42Earlier quoted context omitted.
Also note that it's not necessarily safe to read these documents even if you don't intend on executing embedded code. For example, reading from pdfs uses poppler, which has had a few CVEs that could result in arbitrary code execution, mostly around image decoding. https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=poppler (No shade to poppler intended, just the first tool on the list I looked at.)
Couldn't or shouldn't each parser be run in a container with systemd-nspawn or LXC or another container runtime? (Even if all it's doing is reading a file format into process space as NX data not code as the current user)
Executable-space protection > Limitations mentions JITs and ROP: https://en.wikipedia.org/wiki/Executable-space_protection
mprotect(), VirtualAlloc[Ex] and VirtualProtect[Ex],
"NX bit: does it protect the stack?" https://security.stackexchange.com/questions/47807/nx-bit-do...
Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.
#43Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…
Might be iffier in OCR mode: it seems to use Tesseract, which is known to have issues recognising ligatured text.
Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.
#44It's somewhat similar to 'recoll' in its functionality, only with recoll you need to index everything before search. It even uses the same approach of using third-party software like poppler for extracting the contents.
Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.
#45Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…
Chrome mobile on Android does render Office with what looks like at least an fi ligature for me (it should use an ffi one but still).
Maybe it depends on the font?
Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.
#46Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…
Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.
#47https://github.com/junegunn/fzf/blob/master/ADVANCED.md#ripg...
Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.
#48Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…
The (standard) ripgrep regex engine has full unicode support. My reading of that is that it should handle such equivalences like matching the decomposed version.
To add more color to this, the precise details of what "Unicode support" means are documented here: https://github.com/rust-lang/regex/blob/master/UNICODE.md
In effect, all of UTS#18 Level 1 is covered with a couple caveats. This is already a far cry better than most regex engines, like PCRE2, which has limited support for properties and no way to do subtraction or intersection of character classes. Other regex engines, like Javascript, are catching up. While UTS#18 Level 1 make ripgrep's Unicode support better than most, it does not make it the best. The third party Python `regex` library, for example, has very good support, although it is not especially fast[1].
Short of building UTS#18 2.1[2] support into the regex engine (unlikely to ever happen), it's likely ripgrep could offer some sort of escape hatch. Perhaps, for example, an option to normalize all text searched to whatever form you want (nfc, nfd, nfkc or nfkd). The onus would still be on you to write the corresponding regex pattern though. You can technically do this today with ripgrep's `--pre` flag, but having something built-in might be nice. Indeed, if you read UTS#18 2.1, you'll note that it is self-aware about how difficult matching canonical equivalents is, and essentially suggests this exact work-around instead. The problem is that it would need to be opt-in and the user would need to be aware of the problem in the first place. That's... a stretch, but probably better than nothing.
[1]: https://github.com/BurntSushi/rebar?tab=readme-ov-file#summa...
[2]: https://unicode.org/reports/tr18/#Canonical_Equivalents
Re: Rga: Ripgrep, but also search in PDFs, E-Books, Office documents, zip, etc.
#49Anyone know how it handles ligatures? Depending on font and tooling the word "fish" may end up in various docs as the glyphs [fi, s, h] or [f, i, s, h]. According to a quick check against /usr/share/dict/words "fi" occurs in about 1.5% of words and "fl" occurs in about 1%. There are other ligatures that sometimes occur but those are the most common in English I believe. I don't have any sense of how common ligature us…
It's much more common in PDFs than it is on the web, at least when the underlying plaintext is concerned.