Live data from Hacker News

Weird Lexical Syntax

justine.lol

91–100 of 234 posts

Re: Weird Lexical Syntax

#91
post #22

Some random things that the author seem to have missed: > but TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings Many more languages support that: C# $"{x} plus {y} equals {x + y}" Python f"{x} plus {y} equals {x + y}" JavaScript `${x} plus ${y} equals ${x + y}` Ruby "#{x} plus #{y} equals #{x + y}" Shell "$x plus $y equals $…

> actual code being embedded inside strings

My view on this is that it shouldn’t be interpreted as code being embedded inside strings, but as a special form of string concatenation syntax. In turn, this would mean that you can nest the syntax, for example:

    "foo { toUpper("bar { x + y } bar") } foo"
The individual tokens being (one per line):

    "foo {
    toUpper
    (
    "bar {
    x
    +
    y
    } bar"
    )
    } foo"
If `+` does string concatenation, the above would effectively be equivalent to:

    "foo " + toUpper("bar " + (x + y) + " bar") + " foo"
I don’t know if there is a language that actually works that way.

Re: Weird Lexical Syntax

#92
post #36

> Every C programmers (sic) knows you can't embed a multi-line comment in a multi-line comment. And every Standard ML programmer might find this to be a surprising limitation. The following is a valid Standard ML program: (* (* Nested (**) *) comment *) val _ = print "hello, world\n" Here is the output: $ sml Given how C was considered one of the "expressive" languages when it arrived, it's curious that nested commen…

Lexing nested comments requires maintaining a stack (or at least a nesting-level counter). That wasn’t traditionally seen as being within the realm of lexical analysis, which would only use a finite-state automaton, like regular expressions.

Re: Weird Lexical Syntax

#93

I don't understand why you wouldn't use Tree Sitter's syntax highlighting for this. I mean it's not going to be as fast but that clearly isn't an issue here. Is this a "no third party dependencies" thing?

I don't want to require everyone who builds llamafile from source need to install rust. I don't even require that people install the gperf command, since I can build gperf as a 700kb actually portable executable and vendor it in the repo. Tree sitter I'd imagine does a really great highly precise job with the languages it supports. However it appears to support fewer of them than I am currently. I'm taking a breadth first approach to syntax highlighting, due to the enormity of languages LLMs understand.

Re: Weird Lexical Syntax

#94
post #64

> The languages I decided to support are Ada, Assembly, BASIC, C, C#, C++, COBOL, CSS, D, FORTH, FORTRAN, Go, Haskell, HTML, Java, JavaScript, Julia, JSON, Kotlin, ld, LISP, Lua, m4, Make, Markdown, MATLAB, Pascal, Perl, PHP, Python, R, Ruby, Rust, Scala, Shell, SQL, Swift, Tcl, TeX, TXT, TypeScript, and Zig. A few (admittedly silly) questions about the list: 1. Why no Erlang, Elixir, or Crystal? Erlang appears to be…

Tiobes's index is quite literally worthless, especially with regards to its stated purpose, let alone as a general point of orientation.

I'd wish that purple would stop lending it any credibility.

Re: Weird Lexical Syntax

#95
post #79
post #71

Earlier quoted context omitted.

Except that text inside #if 0 still has to lex correctly. (unifdef has some evil code to support using C-style preprocessor directives with non-C source, which mostly boils down to ignoring comments. I don’t recommend it!)

> Except that text inside #if 0 still has to lex correctly. Are you sure? I just tried on godbolt and that’s not true with gcc 14.2. I’ve definitely put syntax errors intentionally into #if 0 blocks and had it compile. Are you thinking of some older version or something? I thought the pre-processor ran before the lexer since always…

There are three (relevant) phases (see “translation phases” in section 5 of the standard):

• program is lexed into preprocessing tokens; comments turn into whitespace

• preprocessor does its thing

• preprocessor tokens are turned into proper tokens; different kinds of number are disambiguated; keywords and identifiers are disambiguated

If you put an unclosed comment inside #if 0 then it won’t work as you might expect.

Re: Weird Lexical Syntax

#96
post #22

Some random things that the author seem to have missed: > but TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings Many more languages support that: C# $"{x} plus {y} equals {x + y}" Python f"{x} plus {y} equals {x + y}" JavaScript `${x} plus ${y} equals ${x + y}` Ruby "#{x} plus #{y} equals #{x + y}" Shell "$x plus $y equals $…

Is this a bash-ism?

    "$x plus $y equals $((x+y))"

Re: Weird Lexical Syntax

#97

Earlier quoted context omitted.

I am surprised to hear that structural editing has been a hurdle for you, and I think I can offer a piece of advice. I also used to be terrified by its apparent complexity, but later found out that one just needs to use parinfer and to know key bindings for only three commands: slurp, barf, and raise. With just these four things you will be 95% there, enjoying the fruits of paredit without any complexity — all the re…

Thanks very much for the advice, it's timely. It's not so much the editing itself but the unfamiliarity of the ecosystem. It seems it's a square-peg I've been crafting a round hole of habits for it: I guess I should use emacs? How to even configure it such that these actions are available? Or maybe I should write a plugin for helix so that I can be in a familiar environment. Oh, but the helix plugin language is a sch…

To be fair, I am not a "lisper" and I don't know Emacs at all. I am just a Clojure enjoyer who uses IntelliJ + Cursive with its built-in parinfer/paredit.

Re: Weird Lexical Syntax

#98
post #91
post #22

Some random things that the author seem to have missed: > but TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings Many more languages support that: C# $"{x} plus {y} equals {x + y}" Python f"{x} plus {y} equals {x + y}" JavaScript `${x} plus ${y} equals ${x + y}` Ruby "#{x} plus #{y} equals #{x + y}" Shell "$x plus $y equals $…

> actual code being embedded inside strings My view on this is that it shouldn’t be interpreted as code being embedded inside strings, but as a special form of string concatenation syntax. In turn, this would mean that you can nest the syntax, for example: "foo { toUpper("bar { x + y } bar") } foo" The individual tokens being (one per line): "foo { toUpper ( "bar { x + y } bar" ) } foo" If `+` does string concatenati…

Indeed in some of the listed languages you can nest it like that, but in others (e.g. Python) you can't. I would guess they deliberately don't want to enable that and it's not a problem in their parser or something.

Re: Weird Lexical Syntax

#99
post #76

Earlier quoted context omitted.

I think 3 is not only a natural state, but the best state. I don’t think we can have 1 language that satisfies the needs of all people who write code, and thus, we can’t have 1 syntax that does that either. 3 seems the only sensible solution to me, and we have it.

I dunno, here in 3 the hardest part of learning a language has little to do with the language itself and more to do with the ecosystem of tooling around that language. I think we could more easily get on to the business of using the right language for the job if more of that tooling was shared. If each language, for instance did not have it's own package manager, its own IDE, its own linters and language servers all…

I really like the Linux package managers. If you're going to write an application that will run on some system, it's better to bake dependencies into it. And with virtualization and containerization, the system is not tied to a physical machine. I've been using containers (incus) more and more for real development purposes as I can use almost the same environment to deploy. I don't care much about the IDE, but I'm glad we have LSP, Tree-sitter, and DAP. The one thing I do not like is the proliferation of tooling version manager (NVM,..) instead of managing the environment itself (tied to the project).

Re: Weird Lexical Syntax

#100
The author may have missed that lexing C is actually context-sensitive, i.e. you need a symbol table: https://en.wikipedia.org/wiki/Lexer_hack

Of course, for syntax highlighting this is only relevant if you want to highlight the multiplication operator differently from the dereferencing operator, or declarations differently from expressions.

More generally, however, I find it useful to highlight (say) types differently from variables or functions, which in some (most?) popular languages requires full parsing and symbol table information. Some IDEs therefore implement two levels of syntax highlighting, a basic one that only requires lexical information, and an extended one that kicks in when full grammar and type information becomes available.

Post reply on HN