Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weird Lexical Syntax

justine.lol

Post reply on HN
Print view View original

71–80 of 234 posts

Re: Weird Lexical Syntax

#71

Post by fanf2 »

gsliepen wrote: post #58
susam wrote: post #36

> Every C programmers (sic) knows you can't embed a multi-line comment in a multi-line comment. And every Standard ML programmer might find this to be a surprising limitation. The following is a valid Standard ML program: (* (* Nested (**) *) comment *) val _ = print "hello, world\n" Here is the output: $ sml Given how C was considered one of the "expressive" languages when it arrived, it's curious that nested commen…

Well there is one way to nest comments in C, and that's by using #if 0: #if 0 This is a #if 0 nested comment! #endif #endif

Except that text inside #if 0 still has to lex correctly.

(unifdef has some evil code to support using C-style preprocessor directives with non-C source, which mostly boils down to ignoring comments. I don’t recommend it!)

Re: Weird Lexical Syntax

#72

Post by fanf2 »

pansa2 wrote: post #15

> TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings. So to highlight a string, one must count curly brackets and maintain a stack of parser states. Presumably this is also true in Python - IIRC the brace-delimited fields within f-strings may contain arbitrary expressions. More generally, this must mean that the lexical gramm…

Complicated interpolation can be lexed as a regular language if you treat strings as three separate lexical things, eg in JavaScript template literals there are,

    `stuff${
    }stuff${
    }stuff`
so the ${ and } are extra closing and opening string delimiters, leaving the nesting to be handled by the parser.

You need a lexer hack so that the lexer does not treat } as the start of a string literal, except when the parser is inside an interpolation but all nested {} have been closed.

Re: Weird Lexical Syntax

#73

Post by dilap »

notsylver wrote: post #66
dilap wrote: post #25

Earlier quoted context omitted.

llama.cpp does support something like this -- you can give it a grammar which restricts the set of available next tokens that are sampled over so in theory you could notice "```python" or whatever and then start restricting to valid python code. (in least in theory, not sure how feasible/possible it would be in practice w/ their grammar format.) for code i'm not sure how useful it would be since likely any model that…

yeah, ive used llama.cpp grammars before, which is why i was thinking about it. i just think it'd be cool for llamafile to do basically that, but with included defaults so you could eg, require JSON output. it could be cool for prototyping or something. but i dont think that would be too useful anyway, most of the time i think you would want to restrict it to a specific schema, so i can only see it being useful for s…

gotchya. i do think that is a cool idea actually -- LLMs tiny enough to do useful things with formally structured output but not big enough to nail the structure ~100% is probably not an empty set.

Re: Weird Lexical Syntax

#75

Post by masfuerte »

dahart wrote: post #65
susam wrote: post #62

Earlier quoted context omitted.

> ML didn’t have single-line comments, so same level of surprising limitation. It is not quite clear to me why the lack of single-line comments is such a surprising limitation. After all, a single-line block comment can easily serve as a substitute. However, there is no straightforward workaround for the lack of nested block comments. > I’ve never heard someone refer to C as “expressive”, but maybe it was in 1972 whe…

Fair enough. From my perspective, lack of single line comments is a little surprising because most other languages had it at the time (1973, when ML was introduced). Lack of nested comments doesn’t seem surprising, because it isn’t an important feature for a language, and because most other languages did not have it at the time (1972, when C was introduced). I can imagine both pro and con arguments for supporting nes…

AFAIK, C didn't get single line comments until C99. They were a C++ feature originally.

Re: Weird Lexical Syntax

#76

Post by dartos »

__MatrixMan__ wrote: post #68
mqus wrote: post #46

Earlier quoted context omitted.

Having a simple syntax might be fine for computers but syntax is mainly designed to be read and written by humans. Having a simple one like lisp then just makes syntactic discussions a semantic problem, just shifting the layers. And I think an complex syntax is far easier to read and write than a simple syntax with complex semantics. You also get a faster feedback loop in case the syntax of your code is wrong vs the…

Jury's out re: whether I feel this in my gut. Need more time with the lisps for that. But re: cognitive load maybe it goes like: 1. 1 language to rule them all, fancy syntax 2. Many languages, 1 simple syntax to rule them all 3. Many languages and many fancy syntaxes Here in the wreckage of the tower of babel, 1. isn't really on the table. But 2. might have benefits because the inhumanity of the syntax need only be c…

I think 3 is not only a natural state, but the best state.

I don’t think we can have 1 language that satisfies the needs of all people who write code, and thus, we can’t have 1 syntax that does that either.

3 seems the only sensible solution to me, and we have it.

Re: Weird Lexical Syntax

#77

Post by tom_ »

kragen wrote: post #19
keybored wrote: post #9

Meanwhile NeoVim doesn’t syntax highlight my commit message properly if I have messed with "commit cleanup" enough. The comment character in Git commit messages can be a problem when you insist on prepending your commits with some "id" and the id starts with `#`. One suggestion was to allow backslash escapes in commit messages since that makes sense to a computer scientist.[1] But looking at all of this lexical stuff…

> Maybe it was hard to use backtick in the 70’s and 80’s, but today[2] you could use backtick to start a string and a single quote to end it. That's how quoting works by default in m4 and TeX, both defined in the 70s. Unfortunately Unicode retconned the ASCII apostrophe character ' to be a vertical line, maybe out of a misguided deference to Microsoft Windows, and now we all have to suffer the consequences. (Unless w…

See also: https://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html

Re: Weird Lexical Syntax

#78

Post by dahart »

masfuerte wrote: post #75
dahart wrote: post #65

Earlier quoted context omitted.

Fair enough. From my perspective, lack of single line comments is a little surprising because most other languages had it at the time (1973, when ML was introduced). Lack of nested comments doesn’t seem surprising, because it isn’t an important feature for a language, and because most other languages did not have it at the time (1972, when C was introduced). I can imagine both pro and con arguments for supporting nes…

AFAIK, C didn't get single line comments until C99. They were a C++ feature originally.

Oh wow, I didn’t remember that, and I did start writing C before 99. I stand corrected. I guess that is a little surprising. ;)

Is true that many languages had single line comments? Maybe I’m forgetting more, but I remember everything else having single line comments… asm, basic, shell. I used Pascal in the 80s and apparently forgot it didn’t have line comments either?

Re: Weird Lexical Syntax

#79

Post by dahart »

fanf2 wrote: post #71
gsliepen wrote: post #58

Earlier quoted context omitted.

Well there is one way to nest comments in C, and that's by using #if 0: #if 0 This is a #if 0 nested comment! #endif #endif

Except that text inside #if 0 still has to lex correctly. (unifdef has some evil code to support using C-style preprocessor directives with non-C source, which mostly boils down to ignoring comments. I don’t recommend it!)

> Except that text inside #if 0 still has to lex correctly.

Are you sure? I just tried on godbolt and that’s not true with gcc 14.2. I’ve definitely put syntax errors intentionally into #if 0 blocks and had it compile. Are you thinking of some older version or something? I thought the pre-processor ran before the lexer since always…

Re: Weird Lexical Syntax

#80

Post by masfuerte »

dahart wrote: post #78
masfuerte wrote: post #75

Earlier quoted context omitted.

AFAIK, C didn't get single line comments until C99. They were a C++ feature originally.

Oh wow, I didn’t remember that, and I did start writing C before 99. I stand corrected. I guess that is a little surprising. ;) Is true that many languages had single line comments? Maybe I’m forgetting more, but I remember everything else having single line comments… asm, basic, shell. I used Pascal in the 80s and apparently forgot it didn’t have line comments either?

That's my recollection, that most languages had single line comments. Some had multi-line comments but C++ is the first I remember having syntaxes for both. That said, I'm not terribly familiar with pre-80s stuff.
Post reply on HN