Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Adding lookbehinds to rust-lang/regex

systemf.epfl.ch

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Adding lookbehinds to rust-lang/regex

#21

Post by librasteve »

burntsushi wrote: post #13
librasteve wrote: post #11

It’s odd to see such a widely adopted language as Rust only just getting some regex basics. Whereas Raku ( https://raku.org ) has made a strong forward step in regex syntax over PCRE, made by the same language designer with implementation of modern unicode savvy features like Grapheme and Diacritic handling that are essential to building consistent code to handle multilingual needs. say "Cool" ~~ / * /; # ｢Cool｣ say…

It's not only just getting some "regex basics." The `fancy-regex` crate has provided look-behind for years. The OP is about adopting look-behind to the linear time guarantee required by the `regex` crate. My main focus for the `regex` crate has been on performance: https://github.com/BurntSushi/rebar How does Raku's regex performance compare to Perl?

I stand corrected on that - I was responding to the headline and did not appreciate that Rust has had library support beforehand. (That said, having regex around in different standard vs. crate options is not necessarily the ideal).

It's good to have a focus and I agree that Rust is all about performance and stability for a system language.

I haven't seen Raku regex performance benchmarked, but I would be surprised if it beats perl or Rust.

I wouldn't say that Raku is a good choice where speed is the most important consideration since it is a scripting language that runs on a VM with GC. Nevertheless the language syntax includes many features (hyper operators, lazy evaluation to name two) that make it amenable to performance optimisation.

Re: Adding lookbehinds to rust-lang/regex

#22

Post by librasteve »

quotemstr wrote: post #16
librasteve wrote: post #11

It’s odd to see such a widely adopted language as Rust only just getting some regex basics. Whereas Raku ( https://raku.org ) has made a strong forward step in regex syntax over PCRE, made by the same language designer with implementation of modern unicode savvy features like Grapheme and Diacritic handling that are essential to building consistent code to handle multilingual needs. say "Cool" ~~ / * /; # ｢Cool｣ say…

This right here is one of the foundational splits in the programming community. This article is all about how cool an _implementation_ is. This comment is about some other engine's cool _syntax_. Deep versus superficial. The two camps can't stand each other.

Speaking on behalf of the superficial camp, I admire the Rust core regex focus on linear performance and I can well believe that it is based on recent theoretical work.

Splitting the regex features between some core ones that meet a DoS standard and some non-core modules that do other "convenience" features makes sense as a trade off for Rust. It would not make sense in a scripting language like Raku where the weight is on coder expressiveness and making it easier / faster to write working code.

I seem to have hit a seam of intense implementation guys - and they are holding their own since they know their stuff.

I think there is room for improvement BOTH with new system language / core performance innovation AND with advancing the PCRE regex syntax (largely unchanged since the 1990s) and merging it seamlessly with standard language support for Grammars.

Re: Adding lookbehinds to rust-lang/regex

#23

Post by masklinn »

librasteve wrote: post #21
burntsushi wrote: post #13

Earlier quoted context omitted.

It's not only just getting some "regex basics." The `fancy-regex` crate has provided look-behind for years. The OP is about adopting look-behind to the linear time guarantee required by the `regex` crate. My main focus for the `regex` crate has been on performance: https://github.com/BurntSushi/rebar How does Raku's regex performance compare to Perl?

I stand corrected on that - I was responding to the headline and did not appreciate that Rust has had library support beforehand. (That said, having regex around in different standard vs. crate options is not necessarily the ideal). It's good to have a focus and I agree that Rust is all about performance and stability for a system language. I haven't seen Raku regex performance benchmarked, but I would be surprised i…

> That said, having regex around in different standard vs. crate options is not necessarily the ideal

What 1: both regex and fancy-regex are crates. Regex is under the rust-lang umbrella but it’s not part of the stdlib.

What 2: having different options is the point of third partly libraries, why would you have a third party library which is the exact same thing as the standard library?

Re: Adding lookbehinds to rust-lang/regex

#24

Post by SteveJS »

burntsushi wrote: post #13
librasteve wrote: post #11

It’s odd to see such a widely adopted language as Rust only just getting some regex basics. Whereas Raku ( https://raku.org ) has made a strong forward step in regex syntax over PCRE, made by the same language designer with implementation of modern unicode savvy features like Grapheme and Diacritic handling that are essential to building consistent code to handle multilingual needs. say "Cool" ~~ / * /; # ｢Cool｣ say…

It's not only just getting some "regex basics." The `fancy-regex` crate has provided look-behind for years. The OP is about adopting look-behind to the linear time guarantee required by the `regex` crate. My main focus for the `regex` crate has been on performance: https://github.com/BurntSushi/rebar How does Raku's regex performance compare to Perl?

I loved discovering that rust has O(n) guardrails on regex! The so-called features that break that constraint are anti-features.

Over the last two weeks I wrote a dialog aware english sentence splitter using Claude code to write rust. The compile error when it stuck lookarounds in one of the regex’s was super useful to me.

Re: Adding lookbehinds to rust-lang/regex

#25

Post by librasteve »

masklinn wrote: post #23
librasteve wrote: post #21

Earlier quoted context omitted.

I stand corrected on that - I was responding to the headline and did not appreciate that Rust has had library support beforehand. (That said, having regex around in different standard vs. crate options is not necessarily the ideal). It's good to have a focus and I agree that Rust is all about performance and stability for a system language. I haven't seen Raku regex performance benchmarked, but I would be surprised i…

> That said, having regex around in different standard vs. crate options is not necessarily the ideal What 1: both regex and fancy-regex are crates. Regex is under the rust-lang umbrella but it’s not part of the stdlib. What 2: having different options is the point of third partly libraries, why would you have a third party library which is the exact same thing as the standard library?

so Rust has no regex in the standard library, basic/fast regex under the rust-lang umbrella in a crate and fancy-regex is a 3rd party crate

not having different options is the point of (batteries included) standard libraries ;-)

Re: Adding lookbehinds to rust-lang/regex

#26

Post by burntsushi »

librasteve wrote: post #25
masklinn wrote: post #23

Earlier quoted context omitted.

> That said, having regex around in different standard vs. crate options is not necessarily the ideal What 1: both regex and fancy-regex are crates. Regex is under the rust-lang umbrella but it’s not part of the stdlib. What 2: having different options is the point of third partly libraries, why would you have a third party library which is the exact same thing as the standard library?

so Rust has no regex in the standard library, basic/fast regex under the rust-lang umbrella in a crate and fancy-regex is a 3rd party crate not having different options is the point of (batteries included) standard libraries ;-)

We (I am on libs-api in addition to authoring the regex crate) specifically eschewed a batteries included standard library. The fact that `regex` was its own thing was the best thing that ever happened to it. It let me iterate on its API independent of the standard library.

Re: Adding lookbehinds to rust-lang/regex

#27

Post by ncruces »

masklinn wrote: post #15
RadiozRadioz wrote: post #7

From a user perspective, this is extremely valuable. What an amazing improvement; unbounded especially. I do hope this would make it into actual RE2 & go. When I use regex, I expect to be able to lookbehind, so I am routinely hit by RE2's limitations in places where it's used. Sometimes the software uses the entire matched string and you can't use non-capturing groups to work around it. I understand go's reasons, ReD…

The authors’ previous article (linked in this one) was about doing this in re2 ( https://systemf.epfl.ch/blog/re2-lookbehinds/ ), and they have a fork with those changes though I don’t know that they have a PR. > the "purism" of RE2 does fly in the face of practicality to an irksome degree It’s not purism tho. There are very practical reasons to want an FA-based engine, and if you compromise that to get additional fe…

I couldn't find the link in that page, but the fork is here, and seems to be up-to-date: https://github.com/GerHobbelt/re2

If you need that from Go, you can probably use that to create a fork of this: https://github.com/wasilibs/go-re2

Re: Adding lookbehinds to rust-lang/regex

#28

Post by shilangyu »

CJefferson wrote: post #2

Great! I enjoyed reading through, and I'm going to come back later and read a little more carefully. If anyone knows (to let me be lazy), is this the same regex engine used by ripgrep? Or is that an independent implementation?

As others have pointed out, the regex engine is the same so the benefits would trickle downstream. For example, VSCode also uses ripgrep and therefore the rust-lang/regex engine.

Re: Adding lookbehinds to rust-lang/regex

#29

Post by burntsushi »

shilangyu wrote: post #28
CJefferson wrote: post #2

Great! I enjoyed reading through, and I'm going to come back later and read a little more carefully. If anyone knows (to let me be lazy), is this the same regex engine used by ripgrep? Or is that an independent implementation?

As others have pointed out, the regex engine is the same so the benefits would trickle downstream. For example, VSCode also uses ripgrep and therefore the rust-lang/regex engine.

ripgrep plugged this gap a long time ago by providing PCRE2 support.

Re: Adding lookbehinds to rust-lang/regex

#30

Post by dmit »

kibwen wrote: post #17
burntsushi wrote: post #13

Earlier quoted context omitted.

It's not only just getting some "regex basics." The `fancy-regex` crate has provided look-behind for years. The OP is about adopting look-behind to the linear time guarantee required by the `regex` crate. My main focus for the `regex` crate has been on performance: https://github.com/BurntSushi/rebar How does Raku's regex performance compare to Perl?

> the linear time guarantee required by the `regex` crate Making sure this line isn't glossed over: the point of the regex crate is that it provides linear-time guarantees for arbitrary regexes, making it safe (within reason) to expose the regex engine to untrusted input without running the risk of trivial DoS. From what I can tell, supporting lookbehinds in such a context is something that researchers have only rece…

> making it safe (within reason) to expose the regex engine to untrusted input

Or even trusted input! https://blog.cloudflare.com/details-of-the-cloudflare-outage...

Post reply on HN