Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weird Lexical Syntax

justine.lol

Post reply on HN
Print view View original

141–150 of 234 posts

Re: Weird Lexical Syntax

#141

Post by Someone »

saghm wrote: post #133
kazinator wrote: post #35

I don't think it's easy to write a good syntax coloring engine like the one in Vim. Syntax coloring has to handle context: different rules for material nested in certain ways. Vim's syntax higlighter lets you declare two kinds of items: matches and regions. Matches are simpler lexical rules, whereas regions have separate expressions for matching the start and end and middle. There are ways to exclude leading and trai…

Naively, I would have assumed that the "correct" way to write a syntax highlighter would be to parse into an AST and then iterate over the tokens and update the color of a token based on the type of node (and maybe just tracking a diff to avoid needing to recolor things that haven't changed). I'm guessing that if this isn't done, it's for efficiency reasons (e.g. due to requiring parsing the whole file to highlight r…

> I would have assumed that the "correct" way to write a syntax highlighter would be to parse into an AST and then […] I'm guessing that if this isn't done, it's for efficiency reasons

It’s not only running time, but also ease of implementation.

A good syntax highlighter should do a decent job highlighting both valid and invalid programs (rationale: in most (editor, language) pairs, writing a program involves going through moments where the program being written isn’t a valid program)

If you decide to use an AST, that means you need to have good heuristics for turning invalid programs into valid ones that best mimic what the programmer intended. That can be difficult to achieve (good compilers have such heuristics, but even if you have such a compiler, chances are it isn’t possible to reuse them for syntax coloring)

If this simpler approach gives you most of what you can get with the AST approach, why bother writing that?

Also, there are languages where some programs can’t be perfectly parsed or syntax colored without running them. For those, you need this approach.

Re: Weird Lexical Syntax

#142

Post by mdaniel »

kragen wrote: post #86
petesergeant wrote: post #57

> Perl also has this goofy convention for writing man pages in your source code The world corpus of software would be much better documented if everywhere else had stolen this from Perl. Inline POD is great.

Perl and Python stole it from Emacs Lisp, though Perl took it further. I'm not sure where Java stole it from, but nowadays Doxygen is pretty common for C code. Unfortunately this results in people thinking that Javadoc and Doxygen are substitutes for actual documentation like the Emacs Lisp Reference Manual, which cannot be generated from docstrings, because the organization of the source code is hopelessly inadequat…

> Emacs Lisp Reference Manual, which cannot be generated from docstrings, because the organization of the source code is hopelessly inadequate for a reference manual.

Well, they're not doing themselves any favors by just willy nilly mixing C with "user-facing" defuns https://emba.gnu.org/emacs/emacs/-/blob/ed1d691184df4b50da6b...>. I was curious if they could benefit from "literate programming" since OrgMode is the bee's knees but not with that style coding they can't

Re: Weird Lexical Syntax

#143

Post by mdaniel »

IshKebab wrote: post #122
jart wrote: post #93

Earlier quoted context omitted.

I don't want to require everyone who builds llamafile from source need to install rust. I don't even require that people install the gperf command, since I can build gperf as a 700kb actually portable executable and vendor it in the repo. Tree sitter I'd imagine does a really great highly precise job with the languages it supports. However it appears to support fewer of them than I am currently. I'm taking a breadth…

I think the Rust component of tree-sitter-highlight is actually pretty small (Tree Sitter generates C for the actual parser). But fair enough - fewer dependencies is always nice, especially in C++ (which doesn't have a modern package manager) and in ML where an enormous janky Python installation is apparently a perfectly normal thing to require.

I somehow thought Conan[1] was the C++ package manager; it's at least partially supported by GitLab, for what that's worth

1: https://docs.conan.io/2/introduction.html

Re: Weird Lexical Syntax

#144

Post by 1vuio0pswjnm7 »

pdw wrote: post #22

Some random things that the author seem to have missed: > but TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings Many more languages support that: C# $"{x} plus {y} equals {x + y}" Python f"{x} plus {y} equals {x + y}" JavaScript `${x} plus ${y} equals ${x + y}` Ruby "#{x} plus #{y} equals #{x + y}" Shell "$x plus $y equals $…

Shell "$x plus $y equals $((x+y))"

Shell "$x plus $y equals $((expr $x + $y))"

Re: Weird Lexical Syntax

#145

Post by IshKebab »

mdaniel wrote: post #143
IshKebab wrote: post #122

Earlier quoted context omitted.

I think the Rust component of tree-sitter-highlight is actually pretty small (Tree Sitter generates C for the actual parser). But fair enough - fewer dependencies is always nice, especially in C++ (which doesn't have a modern package manager) and in ML where an enormous janky Python installation is apparently a perfectly normal thing to require.

I somehow thought Conan[1] was the C++ package manager; it's at least partially supported by GitLab, for what that's worth 1: https://docs.conan.io/2/introduction.html

No, if anything vcpkg is "the C++ package manager", but it's nowhere near pervasive and easy-to-use enough to come close to even Pip. It's leagues away from Cargo, Go, and other actually good PL package managers.

Re: Weird Lexical Syntax

#146

Post by samatman »

keybored wrote: post #9

Meanwhile NeoVim doesn’t syntax highlight my commit message properly if I have messed with "commit cleanup" enough. The comment character in Git commit messages can be a problem when you insist on prepending your commits with some "id" and the id starts with `#`. One suggestion was to allow backslash escapes in commit messages since that makes sense to a computer scientist.[1] But looking at all of this lexical stuff…

There are no problems caused by using unary delimiters for strings, because using paired delimiters for strings doesn't solve the problems unary delimiters create.

By nature, strings contain arbitrary text. Paired delimiters have one virtue over unary: they nest, but this virtue is only evident when a syntax requires that they must nest, and this is not the case for strings. It's but a small victory to reduce the need for some sort of escaping, without eliminating it.

Of the bewildering variety of partial solutions to the dilemma, none fully satisfactory, I consider the `backtick quote' pairing among the worst. Aside from the aesthetic problems, which can be fixed with the right choice of font, the bare apostrophe is much more common in plain text than an unmatched double quote, and the convention does nothing to help.

This comes at the cost of losing a type of string, and backtick strings are well-used in many languages, including by you in your second paragraph. What we would get in return for this loss is, nothing, because `don't' is just as invalid as 'don't' and requires much the same solution. `This is `not worth it', you see', especially as languages like to treat strings as single tokens (many exceptions notwithstanding) and this introduces a push-down to that parse for, again, no appreciable benefit.

I do agree with you about C and character literals, however. The close quote isn't needed and always struck me as somewhat wasteful. 'a is cleaner, and reduces the odds of typing "a" when you mean 'a'.

Re: Weird Lexical Syntax

#147

Post by shawn_w »

weinzierl wrote: post #128
autarch wrote: post #63

Earlier quoted context omitted.

Perl lets you do this too: my $foo = 5; my $bar = 'x'; my $quux = "I have $foo $bar\'s: @{[$bar x $foo]}"; print "$quux\n"; This prints out: I have 5 x's: xxxxx The "@{[...]}" syntax is abusing Perl's ability to interpolate an _array_ as well as a scalar. The inner "[...]" creates an array reference and the outer "@{...}" dereferences it. For reasons I don't remember, the Perl interpreter allows arbitrary code in the…

You also don't need quotes around strings (barewords). So my $bar = x; should give the same result. Good luck with lexing that properly. https://perlmaven.com/barewords-in-perl

If you're writing anything approaching decent perl that won't be accepted.

Re: Weird Lexical Syntax

#148

Post by ufo »

fanf2 wrote: post #67
SonOfLilit wrote: post #51

Justine gets very close to the hairiest parsing issue in any language without encountering it: Perl's syntax is undecidable, because the difference between treating some characters as a comment or as a regex can depend on the type of a variable that is only determined e.g. based on whether a search for a Collatz counterexample terminates, or just, you know, user input. https://perlmonks.org/?node_id=663393 C++ templa…

I think possibly the most hilariously complicated instance of this is in perl’s tokenizer, toke.c (which starts with a Tolkien quote, 'It all comes from here, the stench and the peril.' — Frodo). There’s a function called intuit_more which works out if $var[stuff] inside a regex is a variable interpolation followed by a character class, or an array element interpolation. Its result can depend on whether something in…

Wow. I wonder how that function came to be in the first place. Surely it couldn't have started out that complicated?

Re: Weird Lexical Syntax

#149

Post by [deleted] »

kazinator wrote: post #35

I don't think it's easy to write a good syntax coloring engine like the one in Vim. Syntax coloring has to handle context: different rules for material nested in certain ways. Vim's syntax higlighter lets you declare two kinds of items: matches and regions. Matches are simpler lexical rules, whereas regions have separate expressions for matching the start and end and middle. There are ways to exclude leading and trai…

[deleted]

Re: Weird Lexical Syntax

#150

Post by mdaniel »

IshKebab wrote: post #145
mdaniel wrote: post #143

Earlier quoted context omitted.

I somehow thought Conan[1] was the C++ package manager; it's at least partially supported by GitLab, for what that's worth 1: https://docs.conan.io/2/introduction.html

No, if anything vcpkg is "the C++ package manager", but it's nowhere near pervasive and easy-to-use enough to come close to even Pip. It's leagues away from Cargo, Go, and other actually good PL package managers.

I knew that Microsoft used that on Windows but had no idea it was multi-platform: https://github.com/microsoft/vcpkg/releases/tag/2024.10.21 (MIT, like a lot of their stuff)

Microsoft is such an odd duck, sometimes, but I'm glad to take advantage of their "good years" while it lasts

Post reply on HN