Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

App::Rak – 21st century grep / find / ack / ag / rg on steroids

github.com

Post reply on HN
Print view View original

81–90 of 98 posts

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#81

Post by larodi »

_v7gu wrote: post #63
larodi wrote: post #55

definitely not going to install cursed Raku for this single search lib. perhaps if they had a distributed binary in some byte-code format, but they don't. because forked-Perl-guys were busy figuring how to use all the Unicode and outpace APL in complexity, while others were figuring how to target reasonably large distribution medium (think wasm, .net, and jre if u want). besides, ripgrep IS much more state of the art…

APL is one of the least complex languages I know (simplest being Lisp); there isn’t much syntax to learn and the language doesn’t really have implicit footguns. You just string together function applications and modify some of those functions with adverbs

I meant this -> https://en.wikipedia.org/wiki/Digital_encoding_of_APL_symbol..., not the language construct.

Let me give an allegory - perhaps not exact, but you'll get my point. Much of the mathematics (either discreet, differential, computation geometry, etc) is actually not so difficult to understand when told right... but incidentally, there were some folks back in the day who decided to write it using the whole greek alphabet and then some cryptic symbols in it (like these https://cloford.com/resources/charcodes/utf-8_mathematical.h...). By doing so they effectively formed some weird cryptic cult of the Mathematicians, which eventually repelled many people away from the discipline, and still do.

Later one, some perhaps not so smart people, figured how to express stuff as 'sum', 'product' in for-loops and suddenly the whole world was writing programs while decrying math as complex. even though... programs are logic are math, right?

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#82

Post by librasteve »

burntsushi wrote: post #44

Author of ripgrep here. Very cool! It looks like Rak has a lot of nice features. I will say that getting this thing built took some digging. Could you please add install/build instructions to the README? As it is, I had to go sleuthing through your CI setup to figure it out. On Archlinux, here's what I did: $ yay -S rakudo-bin $ export PATH="/usr/share/perl6/site/bin:$PATH" $ git clone https://github.com/lizmat/App-R…

@burntsushi - thanks for the benchmarks - certainly rg is amazingly fast! I was wondering if somehow rak and rg could be combined to get the benefit of both worlds (eg. use the strong ffi support in raku and rust to interface them) ... but this would only make any sense if you could apply the rg speed boost to raku regex and thus get the syntax benefits too

(sidebar - Larry took to opportunity with perl6/raku to properly reinvent regexes for unicode - there is quite lot to unpack in raku regexes: graphemes management, unicode props as char classes, named regexes, etc.)

soo - I suggest this proposal to anyone looking for a cool rust/unicode project ... extend ripgrep from PCRE/ECMA262 regex 1.0 to embrace the "21st century" regex syntax of raku

some initial reading is here: - https://dev.to/bbkr/utf-8-glyphs-and-graphemes-331b - https://docs.raku.org/language/regexes - https://github.com/edumentab/p6-ecma262regex

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#83

Post by lizmat »

burntsushi wrote: post #79
lizmat wrote: post #77

Earlier quoted context omitted.

- Support for NFG (Normalization Form Grapheme), which e.g. means that you only need to specify 'é' if you want to look for an 'é', and not have to worry about whether the text you're searching in, consists of the single codepoint 'é', or that it has the decomposed version. - support for --ignoremark, which means you 'e' will match any accented 'e', such as éëêèęėē.

Ah yeah that's a good one! It does very likely have enormous implications for performance though. I wonder if I should add it as an opt-in feature to ripgrep. Although its support will be inherently limited in some capacity since character classes will always be limited to matching a single codepoint. (i.e., No UTS#18 Level 2 support.) Have you found folks using these particular Unicode features in practice? I don't…

> It does very likely have enormous implications for performance though.

Well, but that's only one of the reasons why rak is a lot slower. There's something else going on, but I currently don't have the mindset to investigate this deeply.

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#84

Post by burntsushi »

librasteve wrote: post #82
burntsushi wrote: post #44

Author of ripgrep here. Very cool! It looks like Rak has a lot of nice features. I will say that getting this thing built took some digging. Could you please add install/build instructions to the README? As it is, I had to go sleuthing through your CI setup to figure it out. On Archlinux, here's what I did: $ yay -S rakudo-bin $ export PATH="/usr/share/perl6/site/bin:$PATH" $ git clone https://github.com/lizmat/App-R…

@burntsushi - thanks for the benchmarks - certainly rg is amazingly fast! I was wondering if somehow rak and rg could be combined to get the benefit of both worlds (eg. use the strong ffi support in raku and rust to interface them) ... but this would only make any sense if you could apply the rg speed boost to raku regex and thus get the syntax benefits too (sidebar - Larry took to opportunity with perl6/raku to prop…

To be clear, to do this one would likely need to write an entirely new regex engine. ripgrep's default engine uses finite automata. It's not PCRE2 or ECMA. Many of the things in Raku likely can't easily be implemented in finite automata.

ripgrep does have fairly complete UTS#18 Level 1 support already.

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#85

Post by Yizahi »

sgarland wrote: post #80
Yizahi wrote: post #42

There is problem with nice modern Linux tools - they are not available on the generic servers and embedded boxes with you will likely use, or in the image someone else has given you etc. Therefore it is pointless (sorry for the strong term) to learn them, because they simply won't be available for when they are most needed. And learning new tools is not a free process, there are costs, mental and time.

If you control the servers you’ll be working on, you can install the tooling you’d like. If you don’t, you can lobby for it to be installed. Some Postgres servers I work on have massive log files, and due to DDOG indexing retention being short, sometimes it’s required to go sleuthing on the host. While LC_ALL=C speeds up grep quite a bit, it’s still an order of magnitude slower than ripgrep. Luckily, I can add toolin…

Our test servers are upgraded weekly/daily, and embedded devices we are testing are upgraded daily/hourly. It's just not feasible. And embedded devices has an additional constraint for space and/or bureaucracy. I'm still waiting for the feature complete "less" to be added there after like half a year (we have a weird distro with busybox bolted on top). I understand that my case is maybe an outlier, but I've resigned that I only learn what's available in the busybox and basic Debian.

I wish that distributives maintainers would enforce inclusion of the modern tools like Rak in the default install, then they would become the norm across whole industry. Like for example MS includes Powershell regardless of the opinion of people who dislike it. And now you can expect that powershell is available in any modern Win install. In latest Win11 new Terminal is now a permanent feature in every install and you can expect to have it everywhere on Win11. Etc.

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#86

Post by lizmat »

burntsushi wrote: post #84
librasteve wrote: post #82

Earlier quoted context omitted.

@burntsushi - thanks for the benchmarks - certainly rg is amazingly fast! I was wondering if somehow rak and rg could be combined to get the benefit of both worlds (eg. use the strong ffi support in raku and rust to interface them) ... but this would only make any sense if you could apply the rg speed boost to raku regex and thus get the syntax benefits too (sidebar - Larry took to opportunity with perl6/raku to prop…

To be clear, to do this one would likely need to write an entirely new regex engine. ripgrep's default engine uses finite automata. It's not PCRE2 or ECMA. Many of the things in Raku likely can't easily be implemented in finite automata. ripgrep does have fairly complete UTS#18 Level 1 support already.

> likely need to write an entirely new regex engine

Indeed.

Also, it should be noted that the Raku Regex engine is not a state machine as such. The regex "slang" in Raku, is basically just another way to write code. Code that uses some state machine primitives underneath, for sure, but still code.

A grammar in Raku is nothing other than a class in which the methods are codegenned using the regex syntax. But a class nonetheless, which can also have attributes and "proper" methods.

This gives the Raku regex engine the flexibility needed to be able to parse Raku source code, and generate executable bytecode from that.

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#87

Post by librasteve »

burntsushi wrote: post #84
librasteve wrote: post #82

Earlier quoted context omitted.

@burntsushi - thanks for the benchmarks - certainly rg is amazingly fast! I was wondering if somehow rak and rg could be combined to get the benefit of both worlds (eg. use the strong ffi support in raku and rust to interface them) ... but this would only make any sense if you could apply the rg speed boost to raku regex and thus get the syntax benefits too (sidebar - Larry took to opportunity with perl6/raku to prop…

To be clear, to do this one would likely need to write an entirely new regex engine. ripgrep's default engine uses finite automata. It's not PCRE2 or ECMA. Many of the things in Raku likely can't easily be implemented in finite automata. ripgrep does have fairly complete UTS#18 Level 1 support already.

forgive me for assumption that rg does PCRE(2)/ECMA, we had a bit of a chat on raku and UTS#18 here ... https://www.reddit.com/r/rakulang/comments/15ge67k/uts_18_un... ... raku does not attmept to implement that explicitly but (by virtue of being an HLL and having built ins such as Set operators) comes quite close to it at all levels.

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#88

Post by bberrry »

tialaramex wrote: post #39
bberrry wrote: post #27

Earlier quoted context omitted.

If you grep GB+ sized files, you'll love it. At my previous job I often ended up searching raw log files sized around 5-8GB. Regular grep would grind away for 50-60 seconds, while ripgrep returned the same results in 250ms. I was shocked the first time I used it and have sung it's praises since.

If you can afford it (these products tend to be expensive if they're doing it, and resource hungry if you're doing it, and sometimes both) and you've got logs in files, look at having the logs live in dedicated software. I have used (at different places) Graylog and Splunk, New Relic is also popular. Not that ripgrep isn't awesome, but you could have dedicated indexing and a UI that can be shown to non-technical or a…

Yeah we had ELK for the common logs. This file was a special case that we didn't want indexed and widely accessible because it contained sensitive data. I only occasionally needed to do some digging, so I was fine resorting to ripgrep.

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#89

Post by burntsushi »

librasteve wrote: post #87
burntsushi wrote: post #84

Earlier quoted context omitted.

To be clear, to do this one would likely need to write an entirely new regex engine. ripgrep's default engine uses finite automata. It's not PCRE2 or ECMA. Many of the things in Raku likely can't easily be implemented in finite automata. ripgrep does have fairly complete UTS#18 Level 1 support already.

forgive me for assumption that rg does PCRE(2)/ECMA, we had a bit of a chat on raku and UTS#18 here ... https://www.reddit.com/r/rakulang/comments/15ge67k/uts_18_un... ... raku does not attmept to implement that explicitly but (by virtue of being an HLL and having built ins such as Set operators) comes quite close to it at all levels.

Nice, thank you for the link!

And yeah, I figured Raku might not follow UTS#18 to the letter of the law with respect to syntax, but likely has all the concepts available. Which makes sense given what Raku is.

Re: App::Rak – 21st century grep / find / ack / ag / rg on steroids

#90

Post by sgarland »

Yizahi wrote: post #85
sgarland wrote: post #80

Earlier quoted context omitted.

If you control the servers you’ll be working on, you can install the tooling you’d like. If you don’t, you can lobby for it to be installed. Some Postgres servers I work on have massive log files, and due to DDOG indexing retention being short, sometimes it’s required to go sleuthing on the host. While LC_ALL=C speeds up grep quite a bit, it’s still an order of magnitude slower than ripgrep. Luckily, I can add toolin…

Our test servers are upgraded weekly/daily, and embedded devices we are testing are upgraded daily/hourly. It's just not feasible. And embedded devices has an additional constraint for space and/or bureaucracy. I'm still waiting for the feature complete "less" to be added there after like half a year (we have a weird distro with busybox bolted on top). I understand that my case is maybe an outlier, but I've resigned…

Embedded notwithstanding (I admit that’s a special case where most norms go out the window), you could put the tooling and configs into Ansible plays and run them with the upgrades, or use Puppet to push them continuously.

Regarding distro maintainers, specifically Debian, ripgrep is in Bookworm [0].

[0]: https://packages.debian.org/bookworm/ripgrep

Post reply on HN