Live data from Hacker News

RE#: how we built the fastest regex engine in F#

iev.ee

41–50 of 90 posts

Re: RE#: how we built the fastest regex engine in F#

#41
post #39
post #34

Earlier quoted context omitted.

It uses far less tokens than C#, so watch this space...

Care to explain? Pattern matching, type inference, etc.?

It's all about the goddamned machines.. since F# is terse, they figure agent-generated F# code is cheaper.

Re: RE#: how we built the fastest regex engine in F#

#42
If you claim it's the fastest, how does it compare to one-more-re-nightmare?

- https://github.com/telekons/one-more-re-nightmare

- https://applied-langua.ge/posts/omrn-compiler.html

OMRN is a regex compiler that leverages Common Lisp's compiler to produce optimized assembly to match the given regex. It's incredibly fast. It does omit some features to achieve that, though.

Re: RE#: how we built the fastest regex engine in F#

#43

Earlier quoted context omitted.

> I'm pretty sure that should say "the leftmost start paired with the leftmost end". I’m pretty sure it shouldn’t, that would give you the leftmost shortest match instead of leftmost longest.

As originally written, doesn't it go from the start of the first match to the end of the last match? I feel like I'm missing something.

Right, the explanation seems to be a bit oversimplified, but I don’t think it’s difficult to fix it up: you need to collect non-overlapping starts (with an RTL scan) and ends (with an LTR scan) and zip them together. The non-overlapping matches are the last ones you see before you need to reset the matcher (traverse a failing edge). This feels like it should work.

(I tried to write some pseudocode here but got annoyed dealing with edge cases like zero-length matches at EOF, sorry.)

Re: RE#: how we built the fastest regex engine in F#

#44
post #29

Earlier quoted context omitted.

As originally written, doesn't it go from the start of the first match to the end of the last match? I feel like I'm missing something.

It goes from start of the first match to the longest "alive" end, in practice it will go to a dead state and return after finding the match end. there's an implicit `.*` in front of the first pass but i felt it would've been a long tangent so i didn't want to get into it. so given input 'aabbcc' and pattern `b+`, first reverse pass (using `.*b+`) marks 'aa|b|bcc' the forward pass starts from the first match: 'aa->b|b…

Cheers. I was more confused by how you were doing multiple matches. So I read the paper, which describes the AllEnds algorithm. If I understand correctly, the reverse pass captures all of the match starts and these need to be remembered for the forward pass. Which is what you were showing above, but I didn't follow it.

So, once it gets going, a traditional engine can produce matches iteratively with no further allocation, but RE# requires allocation proportional to the total number of matches. And in return, it's very much faster and much easier to use (with intersection and complement).

Re: RE#: how we built the fastest regex engine in F#

#45
post #39
post #34

Earlier quoted context omitted.

It uses far less tokens than C#, so watch this space...

Care to explain? Pattern matching, type inference, etc.?

Various investigations have found it to be one of the most token efficient statically typed programming languages

https://martinalderson.com/posts/which-programming-languages...

Re: RE#: how we built the fastest regex engine in F#

#47

If you claim it's the fastest, how does it compare to one-more-re-nightmare? - https://github.com/telekons/one-more-re-nightmare - https://applied-langua.ge/posts/omrn-compiler.html OMRN is a regex compiler that leverages Common Lisp's compiler to produce optimized assembly to match the given regex. It's incredibly fast. It does omit some features to achieve that, though.

As a potential user (not the author), what jumps out at me about the two is:

OMRN: No lookaround, eager compilation, can output first match

RE#: No submatches, lazy compilation, must accumulate all matches

Both lookaround and submatch extraction are hard problems, but for practical purposes the lack of lazy compilation feels like it would be the most consequential, as it essentially disqualifies the engine from potentially adversarial REs (or I guess not with the state limit, but then it’s questionable if it actually counts as a full RE engine in such an application).

Re: RE#: how we built the fastest regex engine in F#

#48
post #29

Earlier quoted context omitted.

It goes from start of the first match to the longest "alive" end, in practice it will go to a dead state and return after finding the match end. there's an implicit `.*` in front of the first pass but i felt it would've been a long tangent so i didn't want to get into it. so given input 'aabbcc' and pattern `b+`, first reverse pass (using `.*b+`) marks 'aa|b|bcc' the forward pass starts from the first match: 'aa->b|b…

Cheers. I was more confused by how you were doing multiple matches. So I read the paper, which describes the AllEnds algorithm. If I understand correctly, the reverse pass captures all of the match starts and these need to be remembered for the forward pass. Which is what you were showing above, but I didn't follow it. So, once it gets going, a traditional engine can produce matches iteratively with no further alloca…

Yes, exactly correct

It's also beneficial to merge some of the matching locations into ranges where possible, so when `a*` matches a long sequence of '|a|a|a|a|a|', it can be represented as a range of (0,5), we do this to keep the lookaround internal states smaller in the engine.

Re: RE#: how we built the fastest regex engine in F#

#49
post #41
post #39

Earlier quoted context omitted.

Care to explain? Pattern matching, type inference, etc.?

It's all about the goddamned machines.. since F# is terse, they figure agent-generated F# code is cheaper.

I like to think of F# as concise.

Re: RE#: how we built the fastest regex engine in F#

#50
post #37

Cool stuff. Reminds me of the content you used to see on here all the time before AI took over

I'm sometimes wondering if the AI content here really starts trending organically, or if it is somehow pushed by AI companies.

Not necessarily with bots, just posting a few links in a company Slack with the request for everyone to upvote it from their personal account could be enough.

Post reply on HN