Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres Language Server: Implementing the Parser

supabase.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Postgres Language Server: Implementing the Parser

#21

Post by anarazel »

steinroe wrote: post #3

hey, author here. Thanks for posting it! A bit of background: a few months ago we announced a Postgres language server[0]. A language server adds features like syntax error diagnostic and autocomplete to your editor (vscode, neovim, etc). We have iterated a lot on the parser over the past few months and want to share an update today. the parser is a core piece of any language server that constructs syntax trees from…

Have you investigated generating your parser from postgres' gram.y instead of basically basing it on the bison output?

Re: Postgres Language Server: Implementing the Parser

#22

Post by chatmasta »

steinroe wrote: post #20
chatmasta wrote: post #14

Earlier quoted context omitted.

At Splitgraph we compiled tree-sitter to wasm for Postgres autocomplete. Indeed, one major reason we opted for it is that it has error handling so it can work with incomplete syntax (which is what your code is 90% of the time you're editing it). https://www.splitgraph.com/blog/parsing-pgsql-with-tree-sitt...

thanks for the link, very interesting read! and you are right, libpg_query has its limitations. the idea is to first implement the parser with libpg_query and work around its limitations as good as possible. Since the scan api also returns all tokens for invalid sql, the language server will then have basic features and syntax error diagnostics for invalid statements, and advanced features for valid ones. once the se…

Yes, I think the optimal solution here is a combination of tree-sitter for real time (as-you-type) with a fallback to libpg_query. I mean it's technically the other way around, since libpg_query is preferred when it parses correctly. But yeah I think you inevitably need a combination. Pretty sure TS does similar things in VSCode.

Re: Postgres Language Server: Implementing the Parser

#23

Post by steinroe »

anarazel wrote: post #21
steinroe wrote: post #3

hey, author here. Thanks for posting it! A bit of background: a few months ago we announced a Postgres language server[0]. A language server adds features like syntax error diagnostic and autocomplete to your editor (vscode, neovim, etc). We have iterated a lot on the parser over the past few months and want to share an update today. the parser is a core piece of any language server that constructs syntax trees from…

Have you investigated generating your parser from postgres' gram.y instead of basically basing it on the bison output?

that's a very interesting idea!

for now, our goal is to take the "easy" route with libpg_query and build a language server that provides basic lsp features for invalid sql, and advanced lsp features for valid sql as fast as possible. we then want to go back to the parser and replace the libpg_query-based approach with a more resilient alternative. as of now, the plan is to implement a handwritten recursive-descent statement by statement. will definitely do research to what extend we could leverage gram.y there, especially to potentially fast-track it.

Re: Postgres Language Server: Implementing the Parser

#24

Post by anarazel »

steinroe wrote: post #23
anarazel wrote: post #21

Earlier quoted context omitted.

Have you investigated generating your parser from postgres' gram.y instead of basically basing it on the bison output?

that's a very interesting idea! for now, our goal is to take the "easy" route with libpg_query and build a language server that provides basic lsp features for invalid sql, and advanced lsp features for valid sql as fast as possible. we then want to go back to the parser and replace the libpg_query-based approach with a more resilient alternative. as of now, the plan is to implement a handwritten recursive-descent st…

If some annotations or such would make that easier, it might be possible to do that upstream. Postgres currently has hand-generated code for tab-completion in psql and that's uh, not great (it has gotten large enough that msvc has trouble compiling the file).

Re: Postgres Language Server: Implementing the Parser

#25

Post by mhh__ »

netcraft wrote: post #5

I can't wait for this. Just getting out a great AST would be amazing for things like formatters, but I want to build things like a library that knows everywhere you use a table in a particular way or reference a column, or linting that can help you identify problems with the way you do a join or forgot to apply the right filter for multi-tenant queries.

For a formatter you only need tokens and rules. ASTs are unnecessary unless you want to do transformations at the same time.

Re: Postgres Language Server: Implementing the Parser

#26

Post by steinroe »

epgui wrote: post #19
steinroe wrote: post #13

Earlier quoted context omitted.

in our specific case, we needed something handwritten for the statement-level parser anyways. And the requirements for the LL parser are very simple: extract individual sql statements from a source input. we just compare the next n tokens with a list of tokens from which any statement starts, which is straightforward to implement with a handwritten LL parser. so after all, I would say its a decision based on the spec…

Thanks for your response! I'm a little confused by it though... Maybe I am misunderstanding what you're telling me. I would definitely consider a parser implemented with parser combinators to be "handwritten". The main difference for me is that parser combinators allow you to express a language grammar in a more declarative way, which makes refactoring and correctness verification much simpler. I think what you descr…

sorry, I think I misunderstood your question!

after all, we did not implement a "real" parser. we just use libpg_query, the actual Postgres parser, and work around its limitations as good as possible. The implementation thereby required maximum flexibility. we never define any grammar other than "a select statement starts with a SELECT keyword".

Re: Postgres Language Server: Implementing the Parser

#27

Post by CoastalCoder »

steinroe wrote: post #16
specialist wrote: post #15

Earlier quoted context omitted.

> ...getting the Parser right. It has been a lot of work. True. I have an ANTLR4 grammar for SQL. Am noob. Getting things "correct" was some effort. I have the quixotic goal of accepting multiple dialects. It's a lot of time referencing docs, playing with SQL fiddles, and learning how other grammars work. Now working on performance. Just as tough for noob me. TIL That means removing any potential ambiquities at runti…

that's a huge task to take upon, looking forward to go through it! compared to you, we have gone the "easy" way and use the actual parser from the Postgres server. so no grammar definition and the like. our work was mainly around adapting libpg_query (which is build to parse executable SQL) for our use case.

> use the actual parser from the Postgres server. so no grammar definition and the like.

Doesn't Postgres use Lex/Yacc to describe the grammar?

Re: Postgres Language Server: Implementing the Parser

#28

Post by claytongulick »

kiwicopple wrote: post #2

(i'm on the supabase team) this is an update from the launch here: https://news.ycombinator.com/item?id=37020610 The focus for the past few months has been getting the Parser right. It has been a lot of work. I'll ping Philipp and get him to join the discussion if there are any questions We'd also love more contributors, if this sort of thing is up your alley

Only tangentially related, but if it's of any interest I'm about 90% done with a PEG grammar for the PostgREST query DSL for similar reasons. Happy to share it if it'd be useful.

Re: Postgres Language Server: Implementing the Parser

#29

Post by sjansen »

CoastalCoder wrote: post #27
steinroe wrote: post #16

Earlier quoted context omitted.

that's a huge task to take upon, looking forward to go through it! compared to you, we have gone the "easy" way and use the actual parser from the Postgres server. so no grammar definition and the like. our work was mainly around adapting libpg_query (which is build to parse executable SQL) for our use case.

> use the actual parser from the Postgres server. so no grammar definition and the like. Doesn't Postgres use Lex/Yacc to describe the grammar?

It actually uses Flex/Bison, and does take advantage of the extended features they provide.

Re: Postgres Language Server: Implementing the Parser

#30

Post by atonse »

specialist wrote: post #15
kiwicopple wrote: post #2

(i'm on the supabase team) this is an update from the launch here: https://news.ycombinator.com/item?id=37020610 The focus for the past few months has been getting the Parser right. It has been a lot of work. I'll ping Philipp and get him to join the discussion if there are any questions We'd also love more contributors, if this sort of thing is up your alley

> ...getting the Parser right. It has been a lot of work. True. I have an ANTLR4 grammar for SQL. Am noob. Getting things "correct" was some effort. I have the quixotic goal of accepting multiple dialects. It's a lot of time referencing docs, playing with SQL fiddles, and learning how other grammars work. Now working on performance. Just as tough for noob me. TIL That means removing any potential ambiquities at runti…

If we're all being postgres-specific, isn't there presumably a grammar somewhere in postgres's code that is used to parse the query SQL? Wouldn't you all want to use that same file as a source so it's 100% identical to what Postgres is actually doing?

Genuine question. But maybe the tooling is incompatible.

Also, have you looked at Azure Data Studio? I'm wondering how much you can reuse from the work they did.

Post reply on HN