Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weird Lexical Syntax

justine.lol

Post reply on HN
Print view View original

61–70 of 234 posts

Re: Weird Lexical Syntax

#61

Post by drewr »

mqus wrote: post #46
__MatrixMan__ wrote: post #33

This was a fun read, but it left me a bit more sympathetic to the lisp perspective, which (if I've understood it) is that syntax, being not an especially important part of a language, is more of a hurdle than a help, and should be as simple and uniform as possible so we can focus on other things. Which is sort of ironic because learning how to do structural editing on lisps has absolutely been more hurdle than help s…

Having a simple syntax might be fine for computers but syntax is mainly designed to be read and written by humans. Having a simple one like lisp then just makes syntactic discussions a semantic problem, just shifting the layers. And I think an complex syntax is far easier to read and write than a simple syntax with complex semantics. You also get a faster feedback loop in case the syntax of your code is wrong vs the…

I don't understand your distinction between syntax and semantics. If the semantics are complex, wouldn't that mean the syntax is thus complex?

Re: Weird Lexical Syntax

#62

Post by susam »

dahart wrote: post #43
susam wrote: post #36

> Every C programmers (sic) knows you can't embed a multi-line comment in a multi-line comment. And every Standard ML programmer might find this to be a surprising limitation. The following is a valid Standard ML program: (* (* Nested (**) *) comment *) val _ = print "hello, world\n" Here is the output: $ sml Given how C was considered one of the "expressive" languages when it arrived, it's curious that nested commen…

There are 3 things I find funny about that comment: ML didn’t have single-line comments, so same level of surprising limitation. I’ve never heard someone refer to C as “expressive”, but maybe it was in 1972 when compared to assembly. And what bearing does the comment syntax have on the expressiveness of a language? I would argue absolutely none at all, by definition . :P

> ML didn’t have single-line comments, so same level of surprising limitation.

It is not quite clear to me why the lack of single-line comments is such a surprising limitation. After all, a single-line block comment can easily serve as a substitute. However, there is no straightforward workaround for the lack of nested block comments.

> I’ve never heard someone refer to C as “expressive”, but maybe it was in 1972 when compared to assembly.

I was thinking of Fortran in this context. For instance, Fortran 77 lacked function pointers and offered a limited set of control flow structures, along with cumbersome support for recursion. I know Fortran, with its native support for multidimensional arrays, excelled in numerical and scientific computing but C quickly became the preferred language for general purpose computing.

While very few today would consider C a pinnacle of expressiveness, when I was learning C, the landscape of mainstream programming languages was much more restricted. In fact, the preface to the first edition of K&R notes the following:

"In our experience, C has proven to be a pleasant, expressive and versatile language for a wide variety of programs."

C, Pascal, etc. stood out as some of the few mainstream programming languages that offered a reasonable level of expressiveness. Of course, Lisp was exceptionally expressive in its own right, but it wasn't always the best fit for certain applications or environments.

> And what bearing does the comment syntax have on the expressiveness of a language?

Nothing at all. I agree. The expressiveness of C comes from its grammar, which the language parser handles. Support for nested comments, in the context of C, is a concern for the lexer, so indeed one does not directly influence the other. However, it is still curious that a language with such a sophisticated grammar and parser could not allocate a bit of its complexity budget to support nested comments in its lexer. This is a trivial matter, I know, but I still couldn't help but wonder about it.

Re: Weird Lexical Syntax

#63

Post by autarch »

pdw wrote: post #22

Some random things that the author seem to have missed: > but TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings Many more languages support that: C# $"{x} plus {y} equals {x + y}" Python f"{x} plus {y} equals {x + y}" JavaScript `${x} plus ${y} equals ${x + y}` Ruby "#{x} plus #{y} equals #{x + y}" Shell "$x plus $y equals $…

Perl lets you do this too:

    my $foo = 5;
    my $bar = 'x';
    my $quux = "I have $foo $bar\'s: @{[$bar x $foo]}";
    print "$quux\n";
This prints out:

    I have 5 x's: xxxxx
The "@{[...]}" syntax is abusing Perl's ability to interpolate an _array_ as well as a scalar. The inner "[...]" creates an array reference and the outer "@{...}" dereferences it.

For reasons I don't remember, the Perl interpreter allows arbitrary code in the inner "[...]" expression that creates the array reference.

Re: Weird Lexical Syntax

#64

Post by metadat »

> The languages I decided to support are Ada, Assembly, BASIC, C, C#, C++, COBOL, CSS, D, FORTH, FORTRAN, Go, Haskell, HTML, Java, JavaScript, Julia, JSON, Kotlin, ld, LISP, Lua, m4, Make, Markdown, MATLAB, Pascal, Perl, PHP, Python, R, Ruby, Rust, Scala, Shell, SQL, Swift, Tcl, TeX, TXT, TypeScript, and Zig.

A few (admittedly silly) questions about the list:

1. Why no Erlang, Elixir, or Crystal?

Erlang appears to be just at the author's boundary at #47 on the TIOBE index. https://www.tiobe.com/tiobe-index/

2. What is "Shell"? Sh, Bash, Zsh, Windows Cmd, PowerShell..?

3. Perl but no Awk? Curious why, because Awk is a similar but comparatively trivial language. Widely used, too.

To be fair, Awk, Erlang, and Elixir rank low on popularity. Yet m4, Tcl, TeX, and Zig aren't registered in the top 50 at all.

What's the methodology / criteria? Only things the author is already familiar with?

Still a fun article.

Re: Weird Lexical Syntax

#65

Post by dahart »

susam wrote: post #62
dahart wrote: post #43

Earlier quoted context omitted.

There are 3 things I find funny about that comment: ML didn’t have single-line comments, so same level of surprising limitation. I’ve never heard someone refer to C as “expressive”, but maybe it was in 1972 when compared to assembly. And what bearing does the comment syntax have on the expressiveness of a language? I would argue absolutely none at all, by definition . :P

> ML didn’t have single-line comments, so same level of surprising limitation. It is not quite clear to me why the lack of single-line comments is such a surprising limitation. After all, a single-line block comment can easily serve as a substitute. However, there is no straightforward workaround for the lack of nested block comments. > I’ve never heard someone refer to C as “expressive”, but maybe it was in 1972 whe…

Fair enough. From my perspective, lack of single line comments is a little surprising because most other languages had it at the time (1973, when ML was introduced). Lack of nested comments doesn’t seem surprising, because it isn’t an important feature for a language, and because most other languages did not have it at the time (1972, when C was introduced).

I can imagine both pro and con arguments for supporting nested comments, but regardless of what I think, C certainly could have added support for nested comments at any time, and hasn’t, which suggests that there isn’t sufficient need for it. That might be the entire explanation: not even worth a little complexity.

Re: Weird Lexical Syntax

#66

Post by notsylver »

dilap wrote: post #25
notsylver wrote: post #13

As soon as I saw this was part of llamafile I was hoping that it would be used to limit LLM output to always be "valid" code as soon as it saw the backticks, but I suppose most LLMs don't have problems with that anyway. And I'm not sure you'd want something like that automatically forcing valid code anyway

llama.cpp does support something like this -- you can give it a grammar which restricts the set of available next tokens that are sampled over so in theory you could notice "```python" or whatever and then start restricting to valid python code. (in least in theory, not sure how feasible/possible it would be in practice w/ their grammar format.) for code i'm not sure how useful it would be since likely any model that…

yeah, ive used llama.cpp grammars before, which is why i was thinking about it. i just think it'd be cool for llamafile to do basically that, but with included defaults so you could eg, require JSON output. it could be cool for prototyping or something. but i dont think that would be too useful anyway, most of the time i think you would want to restrict it to a specific schema, so i can only see it being useful for something like a tiny local LLM for code completion, but that would just encourage valid-looking but incorrect code.

i think i just like the idea of restricting LLM output, it has a lot of interesting use cases

Re: Weird Lexical Syntax

#67

Post by fanf2 »

SonOfLilit wrote: post #51

Justine gets very close to the hairiest parsing issue in any language without encountering it: Perl's syntax is undecidable, because the difference between treating some characters as a comment or as a regex can depend on the type of a variable that is only determined e.g. based on whether a search for a Collatz counterexample terminates, or just, you know, user input. https://perlmonks.org/?node_id=663393 C++ templa…

I think possibly the most hilariously complicated instance of this is in perl’s tokenizer, toke.c (which starts with a Tolkien quote, 'It all comes from here, the stench and the peril.' — Frodo).

There’s a function called intuit_more which works out if $var[stuff] inside a regex is a variable interpolation followed by a character class, or an array element interpolation. Its result can depend on whether something in the stuff has been declared as a variable or not.

But even if you ignore the undecidability, the rest is still ridiculously complicated.

https://github.com/Perl/perl5/blob/blead/toke.c#L4502

Re: Weird Lexical Syntax

#68

Post by __MatrixMan__ »

mqus wrote: post #46
__MatrixMan__ wrote: post #33

This was a fun read, but it left me a bit more sympathetic to the lisp perspective, which (if I've understood it) is that syntax, being not an especially important part of a language, is more of a hurdle than a help, and should be as simple and uniform as possible so we can focus on other things. Which is sort of ironic because learning how to do structural editing on lisps has absolutely been more hurdle than help s…

Having a simple syntax might be fine for computers but syntax is mainly designed to be read and written by humans. Having a simple one like lisp then just makes syntactic discussions a semantic problem, just shifting the layers. And I think an complex syntax is far easier to read and write than a simple syntax with complex semantics. You also get a faster feedback loop in case the syntax of your code is wrong vs the…

Jury's out re: whether I feel this in my gut. Need more time with the lisps for that. But re: cognitive load maybe it goes like:

1. 1 language to rule them all, fancy syntax

2. Many languages, 1 simple syntax to rule them all

3. Many languages and many fancy syntaxes

Here in the wreckage of the tower of babel, 1. isn't really on the table. But 2. might have benefits because the inhumanity of the syntax need only be confronted once. The cumulative cost of all the competing opinionated fancy syntaxes may be the worst option. Think of all the hours lost to tabs vs spaces or braces vs whitespace.

Re: Weird Lexical Syntax

#69

Post by _cbsz »

SonOfLilit wrote: post #51

Justine gets very close to the hairiest parsing issue in any language without encountering it: Perl's syntax is undecidable, because the difference between treating some characters as a comment or as a regex can depend on the type of a variable that is only determined e.g. based on whether a search for a Collatz counterexample terminates, or just, you know, user input. https://perlmonks.org/?node_id=663393 C++ templa…

> C++ templates have a similar issue

TIL! I went and dug up a citation: https://blog.reverberate.org/2013/08/parsing-c-is-literally-...

Re: Weird Lexical Syntax

#70

Post by kragen »

keybored wrote: post #40
kragen wrote: post #19

Earlier quoted context omitted.

> Maybe it was hard to use backtick in the 70’s and 80’s, but today[2] you could use backtick to start a string and a single quote to end it. That's how quoting works by default in m4 and TeX, both defined in the 70s. Unfortunately Unicode retconned the ASCII apostrophe character ' to be a vertical line, maybe out of a misguided deference to Microsoft Windows, and now we all have to suffer the consequences. (Unless w…

> That's how quoting works by default in m4 and TeX, both defined in the 70s. Good point. And it was in m4[1] I saw that backtick+apostrophe syntax. I would have probably not thought of that possibility if I hadn’t seen it there. [1] Probably on Wikipedia since I have never used it > Unfortunately Unicode retconned the ASCII apostrophe character ' to be a vertical line, maybe out of a misguided deference to Microsoft…

I'd forgotten about `' in Emacs documentation! That may be influenced by TeX.

C multicharacter literals are unrelated to trigraphs. Trigraphs were a mistake added many years later in the ANSI process.

Post reply on HN