Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weird Lexical Syntax

justine.lol

Post reply on HN
Print view View original

31–40 of 234 posts

Re: Weird Lexical Syntax

#31

Post by ygra »

kragen wrote: post #19
keybored wrote: post #9

Meanwhile NeoVim doesn’t syntax highlight my commit message properly if I have messed with "commit cleanup" enough. The comment character in Git commit messages can be a problem when you insist on prepending your commits with some "id" and the id starts with `#`. One suggestion was to allow backslash escapes in commit messages since that makes sense to a computer scientist.[1] But looking at all of this lexical stuff…

> Maybe it was hard to use backtick in the 70’s and 80’s, but today[2] you could use backtick to start a string and a single quote to end it. That's how quoting works by default in m4 and TeX, both defined in the 70s. Unfortunately Unicode retconned the ASCII apostrophe character ' to be a vertical line, maybe out of a misguided deference to Microsoft Windows, and now we all have to suffer the consequences. (Unless w…

> Unfortunately Unicode retconned the ASCII apostrophe character ' to be a vertical line

Unicode does not precribe the appearance of characters. Although in the code chart¹ it says »neutral (vertical) glyph with mixed usage« (next to »apostrophe-quote« and »single quote«), font vendors have to deal with this mixed usage. And with Unicode the correct quotation marks have their own code points, making it unnecessary to design fonts where the ASCII apostrophe takes their form, but rendering all other uses pretty ugly.

I would regard using ` and ' as paired quotation marks as a hack from times when typographic expression was simply not possible with the character sets of the day.

_________

¹

    0027 ' APOSTROPHE
    = apostrophe-quote (1.0)
    = single quote
    = APL quote
    • neutral (vertical) glyph with mixed usage
    • 2019 ’ is preferred for apostrophe
    • preferred characters in English for paired quotation marks are 2018 ‘ & 2019 ’
    • 05F3 ׳ is preferred for geresh when writing Hebrew
    → 02B9 ʹ modifier letter prime
    → 02BC ʼ modifier letter apostrophe
    → 02C8 ˈ modifier letter vertical line
    → 0301 $́ combining acute accent
    → 030D $̍ combining vertical line above
    → 05F3 ׳ hebrew punctuation geresh
    → 2018 ‘ left single quotation mark
    → 2019 ’ right single quotation mark
    → 2032 ′ prime
    → A78C ꞌ latin small letter saltillo«

Re: Weird Lexical Syntax

#32

Post by Joker_vD »

yen223 wrote: post #26
rererereferred wrote: post #18

In the C# multiquoted strings, how does it know this: Console.WriteLine(""""""); Console.WriteLine(""""""); Are 2 triplequoted empty strings and not one "\nConsole.WriteLine(" sixtuplequoted string?

It's a syntax error! Unterminated raw string literal. https://replit.com/@Wei-YenYen/DistantAdmirableCareware#main...

Ah, so there is no backtracking in lexer for this case. Makes sense.

Re: Weird Lexical Syntax

#33

Post by __MatrixMan__ »

This was a fun read, but it left me a bit more sympathetic to the lisp perspective, which (if I've understood it) is that syntax, being not an especially important part of a language, is more of a hurdle than a help, and should be as simple and uniform as possible so we can focus on other things.

Which is sort of ironic because learning how to do structural editing on lisps has absolutely been more hurdle than help so far, but I'm sure it'll pay off eventually.

Re: Weird Lexical Syntax

#34

Post by pansa2 »

irdc wrote: post #30
pama wrote: post #29

Earlier quoted context omitted.

Interestingly, python f-strings changed their syntax at version 3.12, so highlighting should depend on the version.

It’s just that nesting them arbitrarily is now allowed, right? That shouldn’t matter much for a mere syntax highlighter then. And one could even argue that code that relies on this too much is not really for human consumption.

Also, you can now use the same quote character that encloses an f-string within the {} expressions. That could make them harder to tokenize, because it makes it harder to recognise the end of the string.

Re: Weird Lexical Syntax

#35

Post by kazinator »

I don't think it's easy to write a good syntax coloring engine like the one in Vim.

Syntax coloring has to handle context: different rules for material nested in certain ways.

Vim's syntax higlighter lets you declare two kinds of items: matches and regions. Matches are simpler lexical rules, whereas regions have separate expressions for matching the start and end and middle. There are ways to exclude leading and trailing material from a region.

Matches and regions can declare that they are contained. In that case they are not active unless they occur in a containing region.

Contained matches declare which regions contain them.

Regions declare which other regions they contain.

That's the basic semantic architecture; there are bells and whistles in the system due to situations that arise.

I don't think even Justine could develop that in an interview, other than as an overnight take home.

Re: Weird Lexical Syntax

#36

Post by susam »

> Every C programmers (sic) knows you can't embed a multi-line comment in a multi-line comment.

And every Standard ML programmer might find this to be a surprising limitation. The following is a valid Standard ML program:

  (* (* Nested (**) *) comment *)
  val _ = print "hello, world\n"
Here is the output:

  $ sml
Given how C was considered one of the "expressive" languages when it arrived, it's curious that nested comments were never part of the language.

Re: Weird Lexical Syntax

#38

Post by shawa_a_a »

keybored wrote: post #9

Meanwhile NeoVim doesn’t syntax highlight my commit message properly if I have messed with "commit cleanup" enough. The comment character in Git commit messages can be a problem when you insist on prepending your commits with some "id" and the id starts with `#`. One suggestion was to allow backslash escapes in commit messages since that makes sense to a computer scientist.[1] But looking at all of this lexical stuff…

The comment character is also configurable:

    git config core.commentchar
This is helpful where you want to use use say, markdown to have tidily formatted commit messages make up your pull request body too.

Re: Weird Lexical Syntax

#39

Post by ivanjermakov »

skitter wrote: post #7

Another syntax oddity (not mentioned here) that breaks most highlighters: In Java, unicode escapes can be anywhere, not just in strings. For example, the following is a valid class: class Foo\u007b} and this assert will not trigger: assert // String literals can have unicode escapes like \u000A! "Hello World".equals("\u00E4");

I have never seen this in Java! Is there any use cases where it could be useful?

Re: Weird Lexical Syntax

#40

Post by keybored »

kragen wrote: post #19
keybored wrote: post #9

Meanwhile NeoVim doesn’t syntax highlight my commit message properly if I have messed with "commit cleanup" enough. The comment character in Git commit messages can be a problem when you insist on prepending your commits with some "id" and the id starts with `#`. One suggestion was to allow backslash escapes in commit messages since that makes sense to a computer scientist.[1] But looking at all of this lexical stuff…

> Maybe it was hard to use backtick in the 70’s and 80’s, but today[2] you could use backtick to start a string and a single quote to end it. That's how quoting works by default in m4 and TeX, both defined in the 70s. Unfortunately Unicode retconned the ASCII apostrophe character ' to be a vertical line, maybe out of a misguided deference to Microsoft Windows, and now we all have to suffer the consequences. (Unless w…

> That's how quoting works by default in m4 and TeX, both defined in the 70s.

Good point. And it was in m4[1] I saw that backtick+apostrophe syntax. I would have probably not thought of that possibility if I hadn’t seen it there.

[1] Probably on Wikipedia since I have never used it

> Unfortunately Unicode retconned the ASCII apostrophe character ' to be a vertical line, maybe out of a misguided deference to Microsoft Windows, and now we all have to suffer the consequences. (Unless we're using Computer Modern fonts or other fonts that predate this error, such as VGA font ROM dumps.)

I do think the vertical line looks subpar (and I don’t use it in prose). But most programmers don’t seem bothered by it. :|

> In the 70s and 80s, and into the current millennium on Unix, `x' did look like ‘x’, but now instead it looks like dogshit.

Emacs tries to render it like ‘x’ since it uses backtick+apostrophe for quotes. With some mixed results in my experience.

> Even if you are willing to require a custom font for readability, though, that doesn't solve the problem; you need some way to include an apostrophe in your quoted string!

Aha, I honestly didn’t even think that far. Seems a bit restrictive to not be able to use possessives and contractions in strings without escapes.

> As for end delimiters, C itself supports multicharacter literals, which are potentially useful for things like Macintosh type and creator codes, or FTP commands.

I should have made it clear that I was only considering C-likes and not C itself. A language from the C trigraph days can be excused. To a certain extent.

Post reply on HN