Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weird Lexical Syntax

justine.lol

Post reply on HN
Print view View original

161–170 of 234 posts

Re: Weird Lexical Syntax

#161

Post by epcoa »

layer8 wrote: post #101
panzi wrote: post #98

Earlier quoted context omitted.

Indeed in some of the listed languages you can nest it like that, but in others (e.g. Python) you can't. I would guess they deliberately don't want to enable that and it's not a problem in their parser or something.

Even when nesting is disallowed, my point is that I find it preferable to not view it (and syntax-highlight it) as a “special string” with embedded magic, but as multiple string literals with just different delimiters that allow omitting the explicit concatenation operator, and normal expressions interspersed in between. I think it’s important to realize that it is really just very simple syntactic sugar for normal s…

If it’s treated strictly as simple concatenation syntactic sugar then you are allowing something like print(“foo { func() ); Which seems janky af.

> just very simple syntactic sugar for normal string concatenation.

Maybe. There’s also possibly a string conversion. It seems reasonable to want to disallow implicit string conversion in a concatenation operator context (especially if overloading +) while allowing it in the interpolation case.

Re: Weird Lexical Syntax

#162

Post by chubot »

SonOfLilit wrote: post #51

Justine gets very close to the hairiest parsing issue in any language without encountering it: Perl's syntax is undecidable, because the difference between treating some characters as a comment or as a regex can depend on the type of a variable that is only determined e.g. based on whether a search for a Collatz counterexample terminates, or just, you know, user input. https://perlmonks.org/?node_id=663393 C++ templa…

Yup, bash and GNU Make have the same issue as Perl does, and I mention the C++ issue here too:

Parsing Bash is Undecidable - https://www.oilshell.org/blog/2016/10/20.html

I remember a talk from Larry Wall on Perl 6 (now Raku), where he says this type of thing is a mistake. Raku can be statically parsed, as far as I know.

Re: Weird Lexical Syntax

#163

Post by layer8 »

epcoa wrote: post #110
layer8 wrote: post #91

Earlier quoted context omitted.

> actual code being embedded inside strings My view on this is that it shouldn’t be interpreted as code being embedded inside strings, but as a special form of string concatenation syntax. In turn, this would mean that you can nest the syntax, for example: "foo { toUpper("bar { x + y } bar") } foo" The individual tokens being (one per line): "foo { toUpper ( "bar { x + y } bar" ) } foo" If `+` does string concatenati…

> "foo { … That should probably not be one token. > My view on this is that it shouldn’t be interpreted as code being embedded inside strings I’m not sure exactly what you’re proposing and how it is different. You still can’t parse it as a regular lexical grammar. How does this change how you highlight either? Whatever you call it, to the lexer it is a special string, it has to know how to match it, the delimiters ar…

> > "foo { …

> That should probably not be one token.

It's exactly the point that this is one token. It's a string literal with opening delimiter `"` and closing delimiter `{`, and that whole token itself serves as a kind of opening "brace". Alternatively, you can see `{` as a contraction of `" +`. Meaning, aside from the brace balancing requirement, `"foo {` does the same a `"foo " +` would.

Still alternatively, you could imagine a language that concatenates around string literals by default, similar to how C behaves for sequences of string literals. In C,

    "foo" "bar" "baz"
is equivalent to

    "foobarbaz"
Similarly, you could imagine a language where

    "foo" some_variable "bar"
would perform implicit concatenation, without needing an explicit operator (as in `"foo" + x + "bar"`). And then people might write it without the inner whitespace, as:

    "foo"some_variable"bar"
My point is that

    "foo{some_variable}bar"
is really just that (plus a condition requiring balanced pairs of braces). You can also re-insert the spaces for emphasis:

    "foo{ some_variable }bar"
The fact that people tend to think of `{some_variable}` as an entity is sort-of an illusion.

> How does this change how you highlight either?

You would highlight the `"...{`, `}...{`, and `}..."` parts like normal string literals (they just use curly braces instead of double quotes at one or both ends), and highlight the inner expressions the same as if they weren't surrounded by such literals.

Re: Weird Lexical Syntax

#164

Post by chubot »

IshKebab wrote: post #48

I don't understand why you wouldn't use Tree Sitter's syntax highlighting for this. I mean it's not going to be as fast but that clearly isn't an issue here. Is this a "no third party dependencies" thing?

Have you developed against TreeSitter? Some feedback from people who use it here - https://news.ycombinator.com/item?id=39783471

And here - https://lobste.rs/s/9huy81/tbsp_tree_based_source_processing...

Re: Weird Lexical Syntax

#165

Post by layer8 »

epcoa wrote: post #161
layer8 wrote: post #101

Earlier quoted context omitted.

Even when nesting is disallowed, my point is that I find it preferable to not view it (and syntax-highlight it) as a “special string” with embedded magic, but as multiple string literals with just different delimiters that allow omitting the explicit concatenation operator, and normal expressions interspersed in between. I think it’s important to realize that it is really just very simple syntactic sugar for normal s…

If it’s treated strictly as simple concatenation syntactic sugar then you are allowing something like print(“foo { func() ); Which seems janky af. > just very simple syntactic sugar for normal string concatenation. Maybe. There’s also possibly a string conversion. It seems reasonable to want to disallow implicit string conversion in a concatenation operator context (especially if overloading +) while allowing it in t…

I failed to mention the balancing requirement, that should of course remain. But it's an artificial requirement, so to speak, that is merely there to double-check the programmer's intent. The compiler/parser wouldn't actually care (unlike for an arithmetic expression with unbalanced parentheses, or scope blocks with unbalanced braces), the condition is only checked for the programmer's benefit.

> here’s also possibly a string conversion. It seems reasonable to want to disallow implicit string conversion in a concatenation operator context (especially if overloading +) while allowing it in the interpolation case.

Many languages have a string contenation operator that does implicit conversion to string, while still having a string interpolation syntax like the above. It's kind of my point that both are much more similar to each other than many people seem to realize.

Re: Weird Lexical Syntax

#166

Post by layer8 »

Timwi wrote: post #153
layer8 wrote: post #101

Earlier quoted context omitted.

Even when nesting is disallowed, my point is that I find it preferable to not view it (and syntax-highlight it) as a “special string” with embedded magic, but as multiple string literals with just different delimiters that allow omitting the explicit concatenation operator, and normal expressions interspersed in between. I think it’s important to realize that it is really just very simple syntactic sugar for normal s…

While you're conceptually right, in practice I think it bears mentioning that in C# the two syntaxes compile differently. This is because C#’s target platform, the .NET Framework, has always had a function called `string.Format` that lets you write this: var str = string.Format("{0} is {1} years old.", name, age); When interpolated strings were introduced later, it was natural to have them compile to this instead of…

There's no reason in principle why

    name + " is " + age + " years old."
couldn't compile to exactly the same. (Other than maybe `string.Format` having some additional customizable behavior, I don't know C# that well.)

Re: Weird Lexical Syntax

#167

Post by JadeNB »

autarch wrote: post #121
Izkata wrote: post #118

Earlier quoted context omitted.

> For reasons I don't remember, the Perl interpreter allows arbitrary code in the inner "[...]" expression that creates the array reference. ...because it's an array value? Aside from how the languages handle references, how is that part any different from, for example, this in python: >>> [5 * 'x'] ['xxxxx'] You can put (almost) anything there, as long as it's an expression that evaluates to a value. The resulting v…

I understand that's constructing an array. What's a bit odd is that the interpreter allows you to string interpolate any expression when constructing the array reference inside the string.

> What's a bit odd is that the interpreter allows you to string interpolate any expression when constructing the array reference inside the string.

Why? Surely it is easier for both the language and the programmer to have a rule for what you can do when constructing references to anonymous arrays, without having to special case whether that anonymous array is or is not in a string (or in any one of the many other contexts in which such a construct may appear in Perl).

Re: Weird Lexical Syntax

#168

Post by JadeNB »

emmelaich wrote: post #157
shawn_w wrote: post #147

Earlier quoted context omitted.

If you're writing anything approaching decent perl that won't be accepted.

"use strict" will prevent it and I think strict will be assumed/default soon.

As of Perl 5.12, `use`ing a version (necessary to ensure availability of some of the newer features) automatically implies `use strict`.

https://perldoc.perl.org/strict#HISTORY

Re: Weird Lexical Syntax

#169

Post by jraph »

layer8 wrote: post #109
legobmw99 wrote: post #105

Earlier quoted context omitted.

I’d be shocked if jart didn’t know this, but it seems unlikely that an LLM would generate one of these most vexing parses, unless explicitly asked

Given all the things that were new to the author in the article, I wouldn’t be shocked at all. There’s just a huge number of things to know, or to have come across.

Justine is proficient in C, she is the author of a libc (cosmopolitan) among other things, like Actually Portable Executables [1].

I would expect her to know C quite well, and that's probably an understatement.

[1] https://justine.lol/ape.html

Re: Weird Lexical Syntax

#170

Post by epcoa »

layer8 wrote: post #163
epcoa wrote: post #110

Earlier quoted context omitted.

> "foo { … That should probably not be one token. > My view on this is that it shouldn’t be interpreted as code being embedded inside strings I’m not sure exactly what you’re proposing and how it is different. You still can’t parse it as a regular lexical grammar. How does this change how you highlight either? Whatever you call it, to the lexer it is a special string, it has to know how to match it, the delimiters ar…

> > "foo { … > That should probably not be one token. It's exactly the point that this is one token. It's a string literal with opening delimiter `"` and closing delimiter `{`, and that whole token itself serves as a kind of opening "brace". Alternatively, you can see `{` as a contraction of `" +`. Meaning, aside from the brace balancing requirement, `"foo {` does the same a `"foo " +` would. Still alternatively, you…

> It's exactly the point that this is one token.

Fair enough. The point, as you have acknowledged, being that unlike + you have to treat { specially for balancing (and separately from the “).

> The fact that people tend to think of `{some_variable}` as an entity is sort-of an illusion.

I guess. I just don’t know what being an illusion means formally. It’s not an illusion to the person that has to implement the state machine that balances the delimiters.

> You would highlight the `"...{`, `}...{`, and `}..."` parts like normal string literals (they just use curly braces instead of double quotes at one or both ends), and highlight the inner expressions the same as if they weren't surrounded by such literals

Emacs does it this way FWIW. But I’m not sure how important it is to dictate that the brace can’t be a different color.

In any event, I can agree your design is valid (Kotlin works this way), but I don’t necessarily agree it is any more valid than say how Python does it where there can format specifiers, implicit conversion to string is performed whereas not with concatenation. I’m not seeing the clear definitive advantage of interpolated strings being an equivalent to concatenation vs some other type of method call.

The other detail is order of evaluation or sequencing. String concat may behave differently. Not sure I agree it is wrong, because at the end of the day it is distinct looking syntax. Illusion or not, it looks like a neatly enclosed expression, and concatenation looks like something else. That they might parse, evaluate or behave different isn't unreasonable.

Post reply on HN