Live data from Hacker News

Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

branchfree.org

11–20 of 37 posts

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#11

This appears to be the same algorithm used by ripgrep ( https://github.com/BurntSushi/ripgrep ) for searching when SIMD/AVX is enabled. Specifically, it uses the algorithm Teddy from the library that’s derived from this paper.

Yes, ripgrep (or specifically, the Rust regex crate) takes inspiration from Hyperscan for its SIMD matching.

https://github.com/rust-lang/regex/commit/203c509df9e1dcba61...

Both libraries also rely on a number of techniques from existing published research.

Hyperscan is ruthlessly optimized for x86 platforms, but has dropped support for non-x86 platforms. Rust's regex crate is designed more as a general purpose, portable regular expression implementation, suitable for wide scale use.

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#12
Hyperscan is also used for scanning repos at Github: https://github.blog/2018-10-17-behind-the-scenes-of-github-t.... I have a similar project for scanning repos, https://github.com/zricethezav/gitleaks and plan on adding hyperscan functionality in a future release. Really speeds up scans when trying to match many regexes.

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#13
One of the most interesting ways to handle regex and general parsing IMO is by using derivitives. I'm not sure if this uses any of the ideas from that area, but with derivitive based regex parsing instead of constructing some specific program to parse some language, you parse by instead transforming the original language to be the language of the union of the old language as well as the character just seen.

In addition to being incredibly simple to implement compared to traditional regex and parser generators, I think it is also interesting in that while it is really simple and looks like you are interpreting a language, it is actually powerful enough to parse arbitrary CFGs in O(n^3) with some help from memoization and fixpoints. See http://matt.might.net/articles/parsing-with-derivatives/ for more details.

I think the interesting aspect in terms of performance with something like this is that the derivitive based parsing system is explicitly streaming since it only stores the remaining input string and a regex string. Also, I think you can get around some of the performance issues with constructing these strings by replacing certain parts of the regex with more traditional NFAs/DFAs. In particular, I think this method provides an easy way to use simple equality matching for constant strings, DFAs for Kleene star, NFAs for the rest of the typical NFA features like the or operator and negation, then finally using the derivitive for very complicated features like capturing groups and backreferences.

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#14

One of the most interesting ways to handle regex and general parsing IMO is by using derivitives. I'm not sure if this uses any of the ideas from that area, but with derivitive based regex parsing instead of constructing some specific program to parse some language, you parse by instead transforming the original language to be the language of the union of the old language as well as the character just seen. In additi…

Derivatives are simple to explain, but not entirely straightforward to implement efficiently. I personally prefer the Glushkov construction.

The Glushkov construction (which Hyperscan uses among other things) is a similarly straightforward translation from a regular expression to an epsilon-free NFA with nice properties. There's a very neat paper explaining the construction in the form of a play: https://sebfisch.github.io/haskell-regexp/regexp-play.pdf

On the CFG side, I like to think of Early parsing as analogous to the Glushkov construction. At the very least they have similar properties.

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#15
"Hyperscan Things That Didn’t Make It Into Open Source Hyperscan ... Ports to non-x86 platforms."

This is a big shame, and undercuts the title. Is there any way to release the older stuff so that interested folks can work on it to bring the other arches up to parity? I'm a Power ISA bigot in particular but not having an ARM port seems like a big gap.

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#16
post #6
post #3

This seems tailor-made for deep packet inspection.

Well, it kinda is. AFAIK, that’s exactly what this was made for, and Intel/McAfee was going to create a DPI device using this tech. That never happened, and now Snort uses/will use hyperscan for its pattern matcher

As far as I am aware Arbor is also using hyperscan together with DPDK for DPI/DDoS mitigation

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#17

Author here, in case anyone wants to question, abuse, argue, etc.

I ported Hyperscan to ARM several months ago with help from the following project: https://github.com/nemequ/simde

Would you be interested in working together to get an ARM port that's a bit cleaner and author-approved?

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#18
post #5

Author here, in case anyone wants to question, abuse, argue, etc.

Thank you! I' m very excited to test in my projects ( I found a working Python extension here https://github.com/shenfe/python-hyperscan ) Let me seize the opportunity. I have a problem where I need to match multiple person names (hundreds of thousands) in huge texts. Aho-Corasick works good for exact matches. Could HyperScan works for approximated matches?

Have you tried some existing Python libraries which support fuzzy searching, such as regex and fuzzysearch?

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#19
post #5

Earlier quoted context omitted.

Thank you! I' m very excited to test in my projects ( I found a working Python extension here https://github.com/shenfe/python-hyperscan ) Let me seize the opportunity. I have a problem where I need to match multiple person names (hundreds of thousands) in huge texts. Aho-Corasick works good for exact matches. Could HyperScan works for approximated matches?

Have you tried some existing Python libraries which support fuzzy searching, such as regex and fuzzysearch?

Yes. Both grow the complexity linearly/exponentially based on the number of patterns to be searched.

Re: Hyperscan: A Fast Multi-Pattern Regex Matcher for Modern CPUs

#20

One of the most interesting ways to handle regex and general parsing IMO is by using derivitives. I'm not sure if this uses any of the ideas from that area, but with derivitive based regex parsing instead of constructing some specific program to parse some language, you parse by instead transforming the original language to be the language of the union of the old language as well as the character just seen. In additi…

I'm aware of derivatives. My concern with derivatives (which I think are awesome) is that the combinatorial blow-up that we associate with DFAs is always around the corner.

I've never been a huge fan of putting all my eggs in the the 'I hope it won't explode' basket. It's been a major point of difference with RE2 all this time - IMO you need a high performance pass for things that don't determinize cleanly (even when you are doing lazy determinization).

I would need to see a lot more detail on using derivatives for capturing groups and back-references.

Sorry if I sound closed-minded, but derivatives look like they solve a lot of stuff I already know how to do really well.

Post reply on HN