Weird Lexical Syntax

I could be stretching the definition of "does" but the newfound(?) tree-sitter support in Emacs[1] I believe would allow that since it for sure understands the distinction but I don't possess enough font-lock ninjary to actually, for real, bind a different color to the distinct usages

  /* given foo.c */
  int main() {
    int a, *b;
    a = 5 * 10;
    b = &a;
    printf("a is %d\n", *b);
  }

and then M-x c-ts-mode followed by navigating to each * and invoking M-x treesit-inspect-node-at-point in turn produces, respectively:

  (declaration declarator: (pointer_declarator "*"))

  right: (binary_expression operator: "*")

  arguments: (argument_list (pointer_expression operator: "*"))

1: https://www.emacswiki.org/emacs/Tree-sitter

> Given the neighboring thread where I just learned that the lexer runs before the preprocessor, I’m not sure that would be the outcome.

That is precisely why nested comments would end up breaking the C89 code example I provided above. I elaborate this further below.

> There’s no reason to assume the comment terminator wouldn’t be ignored in strings.

There is no notion of "comment terminator in strings" in C. At any point of time, the lexer is reading either a string or a comment but never one within the other. For example, in C89, C99, etc., this is an invalid C program too:

  #include 

  int main() {
      /* Comment
      printf("hello */ world");
      return 0;
  }

In this case, we wouldn't say that the lexer is "honoring the comment terminator in a string" because, at the point the comment terminator '*/' is read, there is no active string. There is only a comment that looks like this:

      /* Comment
      printf("hello */

The double quotation mark within the comment is immaterial. It is simply part of the comment. Once the lexer has read the opening '/*', it looks for the terminating '*/'. This behaviour would hold even if future C standards were to allow nested comments, which is why nested comments would break the C89 example I mentioned in my earlier HN comment.

> And even today, you can safely write printf("hello // world\n"); without risking a compile error, right?

Right. But it is not clear what this has got to do with my concern that nested comments would break valid C89 programs. In this printf() example, we only have an ordinary string, so obviously this compiles fine. Once the lexer has read the opening quotation mark as the beginning of a string, it looks for an unescaped terminating quotation mark. So clearly, everything until the unescaped terminating quotation mark is a string!

I'm surprised to see that it's allowed to pass a compiled expression to re.match, since the regular expression object has a .match method of its own. To me the fact that the argument is called pattern implies that it's a string, because at the beginning of that chapter, it says, "Both patterns and strings to be searched can be Unicode strings (str) as well as 8-bit strings (bytes). (...) Usually patterns will be expressed in Python code using this raw string notation."

But this ability to pass a compiled regexp rather than a string can't have been an accidental feature, so I don't know why it isn't documented.

Probably it would be good to have an example of invoking re.match with a literal string in the documentation item for re.match that you linked. There are sixteen such examples in the chapter, the first being re.match(r"(\w+) (\w+)", "Isaac Newton, physicist"), so you aren't going to be able to read much of the chapter without figuring out that you can pass a string there, but all sixteen of them come after that section. A useful example might be:

    >>> [s for s in ["", " ", "a ", " a", "aa"] if re.match(r'\w', s)]
    ['a ', 'aa']

It's easy to make manuals worse by adding too much text to them, but in this case I think a small example like that would be an improvement.

As for what type re.compile returns, the section you linked to says, "Compile a regular expression pattern into a regular expression object, which can be used for matching using its match(), search() and other methods, described below." Is your criticism that it doesn't explicitly say that the regular expression object is returned (as opposed to, I suppose, stored in a table somewhere), or that it says "a regular expression object" instead of saying "an re.Pattern object"? Because the words "regular expression object" are a link to the "Regular Expression Objects" section, which begins by saying, "class re.Pattern: Compiled regular expression object returned by re.compile()." To me the name of the class doesn't seem like it adds much value here—to write programs that work using the re module, you don't need to know the name of the class the regular expression objects belong to, just what interface they support.

(It's unfortunate that the class name is documented, because it would be better to rename it to a term that wasn't already defined to mean "a string that can be compiled to a regular expression object"!)

But possibly I've been using the re module long enough that I'm blind to the deficiencies in its documentation?

Anyway, I think documentation questions like this, about gradual introduction, forward references, sequencing, publicly documented (and thus stable) versus internal-only names, etc., are hard to reconcile with the constraints of source code, impossible in most languages. In this case the source code is divided between Python and C, adding difficulty.

Heh. I love Ruby, but, yes, the parser is "interesting", for values of interesting left undefined for its high obscenity content.

It still has bashisms