Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres Language Server: Implementing the Parser

supabase.com

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Postgres Language Server: Implementing the Parser

#41

Post by kiwicopple »

claytongulick wrote: post #28
kiwicopple wrote: post #2

(i'm on the supabase team) this is an update from the launch here: https://news.ycombinator.com/item?id=37020610 The focus for the past few months has been getting the Parser right. It has been a lot of work. I'll ping Philipp and get him to join the discussion if there are any questions We'd also love more contributors, if this sort of thing is up your alley

Only tangentially related, but if it's of any interest I'm about 90% done with a PEG grammar for the PostgREST query DSL for similar reasons. Happy to share it if it'd be useful.

it would definitely be interesting. if it's private and can't share a link, my email is in my profile

Re: Postgres Language Server: Implementing the Parser

#42

Post by ahoka »

steinroe wrote: post #26
epgui wrote: post #19

Earlier quoted context omitted.

Thanks for your response! I'm a little confused by it though... Maybe I am misunderstanding what you're telling me. I would definitely consider a parser implemented with parser combinators to be "handwritten". The main difference for me is that parser combinators allow you to express a language grammar in a more declarative way, which makes refactoring and correctness verification much simpler. I think what you descr…

sorry, I think I misunderstood your question! after all, we did not implement a "real" parser. we just use libpg_query, the actual Postgres parser, and work around its limitations as good as possible. The implementation thereby required maximum flexibility. we never define any grammar other than "a select statement starts with a SELECT keyword".

Parser combinations are very powerful! For example your function whitespace_tokens could be something like:

whitespace_tokens = many1(choice(‘ ‘, ‘\t’, ‘\n’)).map(m -> token(m))

Re: Postgres Language Server: Implementing the Parser

#43

Post by _a_a_a_ »

specialist wrote: post #38
_a_a_a_ wrote: post #37

Earlier quoted context omitted.

Where is the SQLite test suite, please? I'd be very interested. There are already SQL grammars, check https://github.com/antlr/grammars-v4 specifically in here I think https://github.com/antlr/grammars-v4/tree/master/sql I contributed to one of them, and I wrote my own for some personal work. Be warned, it's very involved, very complex and MSSQL is rather ill-defined. Names bracket identifiers) in SQL are bloody awfu…

> Where is the SQLite test suite Here's the scrapped tests: https://github.com/bkiers/sqlite-parser/tree/master/src/test... I didn't find a scrapper. I was cobbling together my own scrapper when I stumbled onto these files. My scrapper is turrible. This SQLite grammar is included in grammars-v4. Looks like a copy vs a fork. I don't know why these tests weren't copied too. > please don't try and write your own grammar…

A, thanks for these tests! That will be really useful for my work.

A couple of thoughts: what do you mean by a 'scrapper'? At first I thought you meant scraper, but I don't think it's that. I haven't come across the term before in this context, can you explain please.

I'm a bit concerned about the times reported for your parsing. It's unfortunately well-known that the python output of antlr is agonisingly slow – what is your output language? (I don't think that's your problem, but if it is then switching to Java output will fix this immediately).

Re: Postgres Language Server: Implementing the Parser

#44

Post by specialist »

_a_a_a_ wrote: post #43
specialist wrote: post #38

Earlier quoted context omitted.

> Where is the SQLite test suite Here's the scrapped tests: https://github.com/bkiers/sqlite-parser/tree/master/src/test... I didn't find a scrapper. I was cobbling together my own scrapper when I stumbled onto these files. My scrapper is turrible. This SQLite grammar is included in grammars-v4. Looks like a copy vs a fork. I don't know why these tests weren't copied too. > please don't try and write your own grammar…

A, thanks for these tests! That will be really useful for my work. A couple of thoughts: what do you mean by a 'scrapper'? At first I thought you meant scraper, but I don't think it's that. I haven't come across the term before in this context, can you explain please. I'm a bit concerned about the times reported for your parsing. It's unfortunately well-known that the python output of antlr is agonisingly slow – what…

Oof. Yes, I meant scraper.

> ...concerned about the times reported

Agreed.

FWIW...

My wall time for this beast (randtest1) is ~55ms.

    SELECT coalesce((select max(11- -19-f-t1.b+a) from t1 where exists(select 1 from t1 where 11-~(d)-c*a*~t1.a-t1.e-t1.e+coalesce((select coalesce((select t1.c from t1 where case (c) when d then e else 11 end=t1.f),t1.d) from t1 where (t1.d)>b), -11) | f+t1.f not in (((c)),b,13))),f) FROM t1 WHERE case when 19+c>=t1.a then t1.c when not case when not exists(select 1 from t1 where +f | b*b*19+19*13-a | case when t1.e not in (t1.f,t1.c,b) then 11 when 17>t1.c then a else e ende) then b when 17=t1.e then b else e endt1.b then a else d end-t1.b=(13)
Whereas simple statements like these are 
    select ((select 1) union (select 1));

    select a between b and c and d;
I need to add profiling metrics to the build, to (proactively) catch performance regressions.

It's fairly easy for ambiguity to sneak in. Mostly because I'm noob. But also because I'm stubbornly using left-recursion and I'm still navigating ANTLR's magical rule rewriting.

> ...python output of antlr is agonisingly slow

Still just using Java.

Edit: Now I'm curious. I'll try ANTLR's Python runtime asap.

My tool is a language translation widget. Input SQL and output templates (vs something like an ORM).

In the future, I'd like to support dynamic languages (stacks) like Python and nodejs/deno. One, just to be a good citizen ("when in Rome..."). But also to try some half-baked notions for supporting dynamic SQL at runtime. Which isn't (yet?) feasible with a compile-time stack like Java.

Thanks for your interest. I put my email in my HN profile. I can send you updates on my work. And once I get to a "beta" release (usable by other people), I intend to have weekly "office hours" on Zoom (or some such).

Re: Postgres Language Server: Implementing the Parser

#45

Post by claytongulick »

kiwicopple wrote: post #41
claytongulick wrote: post #28

Earlier quoted context omitted.

Only tangentially related, but if it's of any interest I'm about 90% done with a PEG grammar for the PostgREST query DSL for similar reasons. Happy to share it if it'd be useful.

it would definitely be interesting. if it's private and can't share a link, my email is in my profile

Sure, still a WIP but it's getting there [1]

[1] https://github.com/superscribe-io/superscribe/blob/developme...

Re: Postgres Language Server: Implementing the Parser

#46

Post by claytongulick »

emptysea wrote: post #36
claytongulick wrote: post #28

Earlier quoted context omitted.

Only tangentially related, but if it's of any interest I'm about 90% done with a PEG grammar for the PostgREST query DSL for similar reasons. Happy to share it if it'd be useful.

Do you have a link to the DSL? Curious to learn more

Yep, posted it in response to sibling comment.

Re: Postgres Language Server: Implementing the Parser

#47

Post by kiwicopple »

claytongulick wrote: post #45
kiwicopple wrote: post #41

Earlier quoted context omitted.

it would definitely be interesting. if it's private and can't share a link, my email is in my profile

Sure, still a WIP but it's getting there [1] [1] https://github.com/superscribe-io/superscribe/blob/developme...

thanks for sharing - we'll check it out
Post reply on HN