Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weird Lexical Syntax

justine.lol

Post reply on HN
Print view View original

101–110 of 234 posts

Re: Weird Lexical Syntax

#101

Post by layer8 »

panzi wrote: post #98
layer8 wrote: post #91

Earlier quoted context omitted.

> actual code being embedded inside strings My view on this is that it shouldn’t be interpreted as code being embedded inside strings, but as a special form of string concatenation syntax. In turn, this would mean that you can nest the syntax, for example: "foo { toUpper("bar { x + y } bar") } foo" The individual tokens being (one per line): "foo { toUpper ( "bar { x + y } bar" ) } foo" If `+` does string concatenati…

Indeed in some of the listed languages you can nest it like that, but in others (e.g. Python) you can't. I would guess they deliberately don't want to enable that and it's not a problem in their parser or something.

Even when nesting is disallowed, my point is that I find it preferable to not view it (and syntax-highlight it) as a “special string” with embedded magic, but as multiple string literals with just different delimiters that allow omitting the explicit concatenation operator, and normal expressions interspersed in between. I think it’s important to realize that it is really just very simple syntactic sugar for normal string concatenation.

Re: Weird Lexical Syntax

#102

Post by Tarean »

panzi wrote: post #98
layer8 wrote: post #91

Earlier quoted context omitted.

> actual code being embedded inside strings My view on this is that it shouldn’t be interpreted as code being embedded inside strings, but as a special form of string concatenation syntax. In turn, this would mean that you can nest the syntax, for example: "foo { toUpper("bar { x + y } bar") } foo" The individual tokens being (one per line): "foo { toUpper ( "bar { x + y } bar" ) } foo" If `+` does string concatenati…

Indeed in some of the listed languages you can nest it like that, but in others (e.g. Python) you can't. I would guess they deliberately don't want to enable that and it's not a problem in their parser or something.

As of python 3.6 you can nest fstrings. Not all formatters and highlighters have caught up, though.

Which is fun, because correct highlighting depends on language version. Haskell has similar problems where different compiler flags require different parsers. Close enough is sufficient for syntax highlighting, though.

Python is also a bit weird because it calls the format methods, so objects can intercept and react to the format specifiers in the f-string while being formatted.

Re: Weird Lexical Syntax

#103

Post by skydhash »

drewr wrote: post #61
mqus wrote: post #46

Earlier quoted context omitted.

Having a simple syntax might be fine for computers but syntax is mainly designed to be read and written by humans. Having a simple one like lisp then just makes syntactic discussions a semantic problem, just shifting the layers. And I think an complex syntax is far easier to read and write than a simple syntax with complex semantics. You also get a faster feedback loop in case the syntax of your code is wrong vs the…

I don't understand your distinction between syntax and semantics. If the semantics are complex, wouldn't that mean the syntax is thus complex?

Most languages' abstract machines expose a very simple API, it's up to the language to add useful constructs to help us write code more efficiently. Languages like Lisp start with a very simple syntax, then add those constructs with the language itself (even though those can be fixed using a standard), others just add it through the syntax. These constructs plus the abstract machine's operations form the semantics, syntax is however the language designer decided to present them.

Re: Weird Lexical Syntax

#104

Post by skydhash »

fanf2 wrote: post #59
pwdisswordfishz wrote: post #14

> Of all the languages, I've saved the best for last, which is Ruby. Now here's a language whose syntax evades all attempts at understanding. TeX with its arbitrarily reprogrammable lexer: how adorable

Lisp reader macros allow you to program its lexer too.

You can basically define a new language with a few lines of code in Racket.

Re: Weird Lexical Syntax

#105

Post by legobmw99 »

layer8 wrote: post #100

The author may have missed that lexing C is actually context-sensitive, i.e. you need a symbol table: https://en.wikipedia.org/wiki/Lexer_hack Of course, for syntax highlighting this is only relevant if you want to highlight the multiplication operator differently from the dereferencing operator, or declarations differently from expressions. More generally, however, I find it useful to highlight (say) types different…

I’d be shocked if jart didn’t know this, but it seems unlikely that an LLM would generate one of these most vexing parses, unless explicitly asked

Re: Weird Lexical Syntax

#106

Post by dahart »

fanf2 wrote: post #95
dahart wrote: post #79

Earlier quoted context omitted.

> Except that text inside #if 0 still has to lex correctly. Are you sure? I just tried on godbolt and that’s not true with gcc 14.2. I’ve definitely put syntax errors intentionally into #if 0 blocks and had it compile. Are you thinking of some older version or something? I thought the pre-processor ran before the lexer since always…

There are three (relevant) phases (see “translation phases” in section 5 of the standard): • program is lexed into preprocessing tokens; comments turn into whitespace • preprocessor does its thing • preprocessor tokens are turned into proper tokens; different kinds of number are disambiguated; keywords and identifiers are disambiguated If you put an unclosed comment inside #if 0 then it won’t work as you might expect…

Ah, I see. You’re right!

Re: Weird Lexical Syntax

#107

Post by layer8 »

ivanjermakov wrote: post #39
skitter wrote: post #7

Another syntax oddity (not mentioned here) that breaks most highlighters: In Java, unicode escapes can be anywhere, not just in strings. For example, the following is a valid class: class Foo\u007b} and this assert will not trigger: assert // String literals can have unicode escapes like \u000A! "Hello World".equals("\u00E4");

I have never seen this in Java! Is there any use cases where it could be useful?

Javac uses the platform encoding [0] by default to interpret Java source files. This means that Java source code files are inherently non-portable. When Java was first developed (and for a long time after), this was the default situation for any kind of plain text files. The escape sequence syntax allows to transform [1] Java source code into a portable (that is, ASCII-only) representation that is completely equivalent to the original, and also to convert it back to any platform encoding.

Source control clients could apply this automatically upon checkin/checkout, so that clients with different platform encodings can work together. Alternatively, IDEs could do this when saving/loading Java source files. That never quite caught on, and the general advice was to stick to ASCII, at least outside comments.

[0] Since JDK 18, the default encoding defaults to UTF-8. This probably also extends to javac, though I haven’t verified it.

[1] https://docs.oracle.com/javase/8/docs/technotes/tools/window...

Re: Weird Lexical Syntax

#108

Post by layer8 »

SonOfLilit wrote: post #51

Justine gets very close to the hairiest parsing issue in any language without encountering it: Perl's syntax is undecidable, because the difference between treating some characters as a comment or as a regex can depend on the type of a variable that is only determined e.g. based on whether a search for a Collatz counterexample terminates, or just, you know, user input. https://perlmonks.org/?node_id=663393 C++ templa…

How could a search for a Collatz counterexample possibly terminate? ;)

Re: Weird Lexical Syntax

#109

Post by layer8 »

legobmw99 wrote: post #105
layer8 wrote: post #100

The author may have missed that lexing C is actually context-sensitive, i.e. you need a symbol table: https://en.wikipedia.org/wiki/Lexer_hack Of course, for syntax highlighting this is only relevant if you want to highlight the multiplication operator differently from the dereferencing operator, or declarations differently from expressions. More generally, however, I find it useful to highlight (say) types different…

I’d be shocked if jart didn’t know this, but it seems unlikely that an LLM would generate one of these most vexing parses, unless explicitly asked

Given all the things that were new to the author in the article, I wouldn’t be shocked at all. There’s just a huge number of things to know, or to have come across.

Re: Weird Lexical Syntax

#110

Post by epcoa »

layer8 wrote: post #91
pdw wrote: post #22

Some random things that the author seem to have missed: > but TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings Many more languages support that: C# $"{x} plus {y} equals {x + y}" Python f"{x} plus {y} equals {x + y}" JavaScript `${x} plus ${y} equals ${x + y}` Ruby "#{x} plus #{y} equals #{x + y}" Shell "$x plus $y equals $…

> actual code being embedded inside strings My view on this is that it shouldn’t be interpreted as code being embedded inside strings, but as a special form of string concatenation syntax. In turn, this would mean that you can nest the syntax, for example: "foo { toUpper("bar { x + y } bar") } foo" The individual tokens being (one per line): "foo { toUpper ( "bar { x + y } bar" ) } foo" If `+` does string concatenati…

> "foo { …

That should probably not be one token.

> My view on this is that it shouldn’t be interpreted as code being embedded inside strings

I’m not sure exactly what you’re proposing and how it is different. You still can’t parse it as a regular lexical grammar.

How does this change how you highlight either?

Whatever you call it, to the lexer it is a special string, it has to know how to match it, the delimiters are materially different than concatenation.

I might be being dense but I’m not sure what’s formally distinct.

Post reply on HN