Solving the regex of madness, and snarky answers on StackOverflow (2019)

Great article (assuming the solution provided works).

I do a lot of parsing in my projects, I find natural text based input vital for power users who don’t to point and click always.

What are some good parsing algorithms, theoretical articles etc to help me become more professional in the parsing tools I write?

I haven't read the rest of the thread, but the article is even wronger than the glib replies there. HTML needn't be well formed. There are adhoc rules which let major browsers parse broken HTML. If you do not follow the spec to the letter you will have your tooling break on input that every browser thinks is acceptable.

Which is why you always use whatever html parsing library comes with your language. There is no simple answer in the thread because there is no simple answer in the real world.

That said, anyone who says:

>It is quite possible and not even that difficult:

    (  
       # match all tags in XHTML but capture only opening tags
         # comment   
      |  # CData section
      | /'""] )* > 
      |  # xml declaration or processing instruction  
      | /'""] )* /> # self-closing tag  
      |  \w+ ) ( "" [^""]* "" | ' [^']* ' | [^>/'""] )* > # opening tag - captured  
      |  # end tag  
    )

Should be seriously mentored by someone.

Does the proposed regular expression really handle embedded script content correctly? From my limited understanding of HTML, pretty much only counts as closing the script contents and everything else is treated as part of the script.

But it should be easy based on this example to include correct HTML tags in the script which the regular expression will emit. Or if you want to recognise HTML tags in the script, you can easily obfuscate construction of in the script using string concatenation.

The only part I agree in this writing is that you don't need to be snarky to be correct. (I'd like to introduce the XY problem of the second kind, where the answerer is so confident that it is the answerer who have missed the actual question.)

Some regexes can recognize a language beyond the regular language. They are typically available in two flavors: recursive references (Perl, Ruby, PCRE) and stackable captures (.NET). They are obscure enough that I would not recommend them, but it is patently false that regular expressions (EDIT: of the practical interest) cannot be recursive.

It is possible to match individual HTML tags with regexes, but it is difficult. It cannot use a bare `\w` or `\s` because both XML/XHTML and HTML5 parsers have peculiar definitions for tag name characters and space characters. For example your `\s` will typically match various Unicode space characters, while only ASCII whitespaces are recognized in tags. There are also several notable exceptions to the parser (and external states termed the "tree construction"), so missing any of them would result in an immediate XSS. If you think you can write a correct regex for HTML tags, my quizzes [1] should make you concerned. Limiting the question to XHTML does alleviate some but not all concerns.

The distinction between recognition and parsing is correct, but parsing doesn't necessarily mean the reconstruction of parse tree. Parsing means the access to constituent nonterminals, which can be used to reconstruct parse tree but also directly used as their own (e.g. calculators). Indeed in most regex implementations you can't extract two or more strings out of each capture (Raku is a notable exception), so you can match against e.g. `(\w+)(?:,(\w+))*` but can't extract a list of comma-separated words with it. Practically speaking this means you can't extract a list of attributes with a single regex, making it unsuitable for HTML parsing anyway.

[1] https://news.ycombinator.com/item?id=26355451