The true power of regular expressions (2012)

While true in principle, writing grammars in regexes is problematic in practice: the syntax for the more advanced features (named submatches, lookahead, backreferences, etc.) is pretty complex, and refactoring the expression means you're working within a string literal, with no help whatsoever from your editor or IDE.

My "go to" solution for parsing (and validating/matching) non-trivial grammars is a library that wraps regexes and allows you to structure the grammar with entities above substrings of a string literal (including arbitrary code for transformations). PyParsing for Python, scala-parser-combinators for Scala, Grammar in Raku, PetitParser in Smalltalk, PEGs in Janet, parser combinators in F#, and so on. These are mostly internal/embedded DSLs, which makes them much easier to use than the typical lexer/parser generators, while giving you all the power to structure and evolve the grammar easily.

For simple grammars, a well-written library adds little overhead over plain regexes. However, grammars rarely stay simple - very often, during the course of development, you find edge cases or the need for extensions. If you started with a structured parser, you're fine: there are specific ways of evolving the grammar, and you can use normal refactoring tools to perform them. If you started with a regex, you quickly end up with a monster regex literal that becomes more brittle and harder to change with each modification.

One important property I look for in parsing libraries is the support for left-recursion. Memoizing/packrat parser generators can handle it gracefully, which is important, because if I'm implementing a published grammar, I want to encode it as closely to the original as possible. For the same reason, I prefer having dedicated tools for associativity and precedence (so that I don't have to invent names for intermediate levels).

TL;DR: yes, regexes are much more expressive than the "regular" in the name would imply, but they still have their limits. For parsing things, it's better to start with something that can work in the simple case fast (so no lex/yacc-style codegen from 2 separate external DSLs), but which also provides enough structure that adding good error handling, extending the grammar, attaching arbitrary code transformations, etc. won't be a big problem later.

I would recommend trying something like PyParsing[1] instead. Libraries like this allow you to compose the parser from language-level entities (object and functions, on top of regex and string literals). This means you can attach comments to those entities naturally within the syntax of the language. You also get much better error reporting out of the box, as well as a well-defined way of attaching transforming code to parts of the parser.

There's a place for simple regexes, but complex regex DSLs (with comments and non-significant whitespace, etc.) are almost always less convenient than simply using your language directly.

[1] https://pyparsing-docs.readthedocs.io/en/latest/HowToUsePypa...

Emacs/Elisp has the rx library: https://www.gnu.org/software/emacs/manual/html_node/elisp/Rx...

You get s-exp-based regex syntax (example for C-style block comments; there are shorter aliases too, e.g. `zero-or-more` can be written as `*`):

    (rx "/*"                    ; Initial /*
        (zero-or-more
         (or (not "*")          ;  Either non-*,
             (seq "*"           ;  or * followed by
                  (not "/"))))  ;     non-/
        (one-or-more "*")       ; At least one star,
        "/")                    ; and the final /

and you have rx-define and rx-let to defined named subforms:

    (rx-let ((comma-separated (item) (seq item (0+ "," item)))
             (number (1+ digit))
             (numbers (comma-separated number)))
      (re-search-forward (rx "(" numbers ")")))

And this is just the regex builder - syntactic sugar - as it still just builds a single regex serialized to a normal string.

I tend to use it everywhere, since it is guaranteed to always properly escape all backslashes (a major pain point in string regexes in Emacs), but it's also useful for building larger regexes from chunks and reusing chunks in multiple related regexes.*

> as might be the case with programmers from Asian countries or others where familiarity with English is not guaranteed

Lots of Asian countries where familiarity with English is assumed in professional contexts.

> So if your primary gateway to programming languages are ones that slightly resemble a human language you are familiar with you may have lots of psychological blocks keeping you from making that final jump to reasoning in J, or APL, or even a DSL like regular expressions.

That raises the interesting possibility that J or APL might be more appealing to non-English speaking countries, or maybe where the dominant languages are not Indo-European (so not similar to English either). I wonder whether there is any evidence of this?

Why could external URLs be a problem? And is it still one if you swap https to hxxps or something? What could go wrong with having a URL as a comment in code?

I put URLs there sometimes and think it's very helpful.