Live data from Hacker News

Open Source SQL Parsers

tokern.io

71–80 of 103 posts

Re: Open Source SQL Parsers

#71
post #18

Earlier quoted context omitted.

At my work, we often parse and rewrite a query before handing it off to SQL Server, because there are a lot of cases where Microsoft misses obvious optimizations. Sometimes there are also optimizations we can do because of things we know at compile time, but don't fit in the type system of SQL. The impact varies all the way from just shaving off 10% of the execution time, to changing some queries from timing out in a…

Hmm? This doesn't make much sense. MSSQL does these already I think (except the first) . - Inlining scalar function calls (less impactful now with Sql Server 2019) write them as table valued funcs and it'll inline them for you (ugly but it works and is easy) . - Removing joins from a query when we know it won't impact the number of records That's just tree pruning. It does that . - Deepening where conditions against…

> This doesn't make much sense. MSSQL does these already I think (except the first)

We tested all of these before taking the time to do the rewriting, and no, either they don't, or they way they did it isn't good enough. You can say I'm lying if you want, I'm not going to spend time arguing about it.

Re: Open Source SQL Parsers

#72
post #5

Not related to parsers, but i find sql syntax so backwards. Listing columns first Then table Then join Then filter Then group bys Then limit The order of operations are out of whack and makes pipeline ing a little hard. I found this to be closer to LINQ way https://github.com/prql/prql I hope in near future databases will come with better query languages...

...and there's more to sql trouble than just troublesome syntax, typical for languages of its generation! 1. Sql is non-modular 2. Non-orthogonal 3. Its parroting of the (beautiful) relational data model is... opinionated at best. The standard is unreadable and not really implementable, one always has to resort to implementation docs. Realistically there are no alternatives, but... sql is no good.

Imagine you need to write a query to answer "what are the cities with more than 4000 inhabitants".

What more natural could there be than to write this as (old 1992 correlated query style)

SELECT cityName FROM Cities C WHERE ((SELECT COUNT() FROM Inhabitants I WHERE I.cityName = C.cityName) > 4000)

Agreed ???

YOU CAN'T DO THIS IN SQL [standard version] !!! You must write

SELECT cityName FROM Cities C WHERE (4000 ) FROM Inhabitants I WHERE I.cityName = C.cityName))

This makes SQL about the only language in the world that allows you to write "aa" in those very same places. Some professional piece of language design.

(Disclaimer : I don't know how matters are in SQL:2016 wrt to this issue, but it drove a former IBM employee who was at the time a member of the SQL standards committee to (a) start work on TTM and (b) leave IBM as soon as he could.)

Re: Open Source SQL Parsers

#73

Earlier quoted context omitted.

I don't mind SQL, really, but I would prefer a query construction API that has the full power of SQL (and more even). At the very least such a thing would not have a SQL injection problem, but also could be used to generate ASTs. Query q = db.from("foo").join("bar").using("id") .select("id","foothing","barthing"); However, the moment you want any non-trivial SQL expression, such an API becomes unwieldy. And yet the n…

That "the API becomes unwieldy" is in fact inevitable. In order to support .where( ) as well as .select( , ascolname) you need a way to pass a parameter of type something-like-expression to the .where() and .select() methods, and lambdas or at least something like them appear to be the right way to do that (e.g. .where((foothing) -> (foothing Plus (assume my .where() example was Java), the host language compiler is n…

> lambdas

Lambdas in the host language won't play well with remote RDBMSes -- you'd have to be able to serialize the lambda and make the serialized form reasonably efficient. I'm skeptical of lambdas for expressions in DB query APIs.

An expression like `foo + bar` has to become `.expr(plus("foo", "bar"))`, except, if you take this to its limit you'll want to use non-string objects to identify column names and other such things, and, again, it quickly becomes unwieldy.

You cover some of this in your comment, so I think we're in agreement:

> That "the API becomes unwieldy" is in fact inevitable.

And yet an API is kinda desirable.

Ultimately I think a query language with no constant literals, only query parameters, compiling to a standard AST that can also be constructed by APIs, may be the best way forward. One could then write in a QL to start with, compile to an AST, serialize into a host language if desired, modify and use that, or always use the QL, or even always use the API, unwieldy though it would be. Then lambdas could actually be in the QL and compiled on the fly as needed:

  Query q = db.query();
  
  q = q.from(...).join(...).using(...)
       .select(...);
  q = q.where(q.expr("foo + bar 
Ok, that snippet has a problem if we want `q` to be a unique pointer. Let's fix it up a bit:

  QueryScope qs;
  Query q = db.query();
  
  q = q.from(...).join(...).using(...)
       .select(...);
  
  /* Get a context of in-scope identifiers, types, ... */
  qs = q.getWhereScope();
  
  /* Now we can compile an expression string */
  q = q.where(qc.expr("foo + bar 
And a hybrid QL + API might look like:

  QueryScope qs;
  Query q = db.parse("SELECT ... FROM ... JOIN ... USING (...)");
  
  /* Get a context of in-scope identifiers, types, ... */
  qs = q.getWhereScope();
  
  /* Now we can compile an expression string */
  q = q.where(qc.expr("foo + bar 

Re: Open Source SQL Parsers

#74

Earlier quoted context omitted.

And if you're set on building an "entire DB engine" then you can just as well go the full mile and go for relational. As opposed to SQL.

I know about TTM etc, but what business value would that bring to be fully relational? Specifically, thanks.

See the "business case for SIRA_PRISE". Imagine how many years of codeshitter-hours you [or, if you're in the DBMS market, your customers] would no longer have to pay for if you could have *ALL* of your [strictly data-related] business rules enforced by a mere *declaration* made by the business analyst c.q. the data administrator (note the absence of 'base' in that term), as opposed to having to rely on those very same codeshitters because there simply ain't no other way for you to enforce same set of said business rules other than to pay that mob for coding all that application-enforced integrity (which they are bound to get wrong because it is fundamentally beyond their ability) or the stored procedures that achieve the same but are still procedural (and is still bound to fail for essentially the same reason).

If you know about TTM, you might also know about "Applied Mathematics for Database Professionals". What I'm referring to is their "execution model 6" becoming possible *WITHOUT* any [procedural form of] coding [by mere programmers].

Another way of saying this is "You can have CREATE ASSERTION if you want to".

Re: Open Source SQL Parsers

#75
post #63

Earlier quoted context omitted.

Linq in c# fixes that.

I doubt that it does. There is way more to "integrating [host] language and queries" than the MIN() of what Micro$oft engineers are (a) capable of understanding and (b) allowed by their own management to put in the products they come up with.

I also doubt it, though I know nothing about Linkq so I'm loathe comment except in so far as what makes the problem hard (see comments above).

Re: Open Source SQL Parsers

#76
post #63

Earlier quoted context omitted.

Linq in c# fixes that.

I doubt that it does. There is way more to "integrating [host] language and queries" than the MIN() of what Micro$oft engineers are (a) capable of understanding and (b) allowed by their own management to put in the products they come up with.

Microsoft is surprisingly good for creating good developer tools and infrastructure.

Re: Open Source SQL Parsers

#77

Earlier quoted context omitted.

I know about TTM etc, but what business value would that bring to be fully relational? Specifically, thanks.

See the "business case for SIRA_PRISE". Imagine how many years of codeshitter-hours you [or, if you're in the DBMS market, your customers] would no longer have to pay for if you could have * ALL* of your [strictly data-related] business rules enforced by a mere * declaration* made by the business analyst c.q. the data administrator (note the absence of 'base' in that term), as opposed to having to rely on those very…

Am aware of sira-prise but not very well. Thanks, will read up (any disclaimer needed here; has sira-prise any link to you?)

I'd appreciate you omitting the codeshitter ad-homs, it undermines your case.

Pretty sure no declarative statement can be made efficient automatically so that remains a dream (though one I will need to look at) so it will kill performance. I too hate procedural enforcements but there seems to be no way round them. Have you got a reliable statement anywhere that says efficient 'create assertion' in sql is possible in general?

AMfDbP book - it's on my reading list already. Thanks for the pointer.

Re: Open Source SQL Parsers

#78

Earlier quoted context omitted.

See the "business case for SIRA_PRISE". Imagine how many years of codeshitter-hours you [or, if you're in the DBMS market, your customers] would no longer have to pay for if you could have * ALL* of your [strictly data-related] business rules enforced by a mere * declaration* made by the business analyst c.q. the data administrator (note the absence of 'base' in that term), as opposed to having to rely on those very…

Am aware of sira-prise but not very well. Thanks, will read up (any disclaimer needed here; has sira-prise any link to you?) I'd appreciate you omitting the codeshitter ad-homs, it undermines your case. Pretty sure no declarative statement can be made efficient automatically so that remains a dream (though one I will need to look at) so it will kill performance. I too hate procedural enforcements but there seems to b…

Link between me and SIRA_PRISE : I am the author.

codeshitter ad-homs : yeah well I know they are. The fact of the matter is the history between SIRA_PRISE and me (and why I did it in the first place) is now almost 20 yrs old, and I know how it's been received, and that's primarily due to (a) how the codeshitters (and the way how they are subject to the Dunning-Kruger effect) have come to dominate the entire industry and (b) how that economic system I too am bound to operate/survive in prevents the managers who control the system from taking "too much" (say, > 0.01%) risks.

And about "Pretty sure no declarative statement can be made efficient automatically so that remains a dream" : there fucking sure ain't no better way to piss me off, not just to the other side of the planet, but to plain outright Mars or Jupiter. You're just too pretty damn sure of yourself. AM4DP makes +- the same statement as you although the authors there still did manage to *NOT* make the same logically flawed inference from "not anybody knowing today how it can be done" to "cannot be done". You apparently fall into the category of people who do make that inference. The article exposing the exact opposite of your convictions was published in Oracle Users Magazine somewhere around 2013, IIRC. I'd need to look up the exact details by now, but it was already an incredible surprise Oracle User Group even wanted to publish on the solution of a problem that Oracle Corporation itself wasn'y (and still isn't, probably due to the power of certain people within that company who have been labeled 'bean keepers') prepared to invest in.

In fact, the only reason I wrote that article in the first place was Toon Koppelaars using that very word 'dream' (which you also used in your reply) to associate the idea with SQL's CREATE ASSERTION, when I already knew it needn't be a 'dream' no longer ... Well, if anyone just wanted to believe, which they don't, you inluded, apparently ... To Toon's credit, although he also used the adjective 'impossible' to qualify that 'dream', he did manage to end his title phrase with a *question* mark. You ended yours with an exclamation mark.

And just for the record, this statement is *NOT* to be interpreted as "every thinkable constraint can be implemented with sub-millisecond violation detection". Sunt certi denique fines, quos ultra citraque nequit consistere rectum. The "fines" in case being the bounds of what algorithmics as it is known today, can do for us. Constraints like "all the people who obtained a degree in mathematics must be paid in the top 5% of salaries" (if anyone ever wanted to formally declare and enforce any rule like that) inevitably takes us into realms of 2nd-order logic that no known algorithm today can guarantee us execution times like the ones we have grown accustomed to from what is supported by SQL systems these days).

As for "do you have a reliable statement" ... I have two answers but neither have the quality of being dependable on the academic level of meaning you might probably want to attach to the words "reliable" and "dependable". The first answer is "No, because the only other person in the entire world that I'm aware of achieving results in the same area is professor Davide Martinenghi and his PhD thesis on the subject, of which I don't even consider myself capable enough to academically assess the equivalence between his results and mine" and the second answer is "No, because that 'reliable' statement would have to be made by me myself and I have nothing(*) but my own implementation to 'prove' it and I'm very well aware that on an academic level, a seemingly working implementation is not a proof".

(*) I did write a paper on the subject and it's been seen by Chris Date (who by proxy answered that he was 'impressed'), by Hugh Darwen, and by Adrian Hudnott. And I will rather take it to my grave with me than disclose it any further.

And about the *scrutinously detailed subject* of your question whether "efficient 'create assertion' in sql is possible in general" : I have grown convinced that *IN SQL*, it isn't. But the reasons are *ENTIRELY* related to SQL's depending on 3VL. I have grown convinced (in fact, for my own conviction, I think I have sufficiently demonstrated) that in some *other* DMl language that *embraces 2VL*, it is *perfectly feasible*. And in fact, I think SIRA_PRISE itself is its own proof, in that respect : *ALL* the rules that SIRA_PRISE imposes as business rules on its users are implemented as mere declared database constraints on the catalog, and are enforced using the *exact* same machinery that also enforces the *user* declared business rules on the *user* databases. When I showed my system to a former colleague of mine who made his entire career in data [base] administration backed by a degree in mathematics, he merely responded that there just ain't no better POC than that.

Re: Open Source SQL Parsers

#79
post #76

Earlier quoted context omitted.

I doubt that it does. There is way more to "integrating [host] language and queries" than the MIN() of what Micro$oft engineers are (a) capable of understanding and (b) allowed by their own management to put in the products they come up with.

Microsoft is surprisingly good for creating good developer tools and infrastructure.

In the perception of the Micro$oft users who are brainwashed with the idea that what Micro$oft does is good for the developers.

(In fact, it might even be outright true. But that's not a guarantee that what the Micro$oft users do with the Micro$oft tools is necessarily also the *BEST* thing for the user of that software product that the Micro$oft users produce.)

Re: Open Source SQL Parsers

#80

Earlier quoted context omitted.

That "the API becomes unwieldy" is in fact inevitable. In order to support .where( ) as well as .select( , ascolname) you need a way to pass a parameter of type something-like-expression to the .where() and .select() methods, and lambdas or at least something like them appear to be the right way to do that (e.g. .where((foothing) -> (foothing Plus (assume my .where() example was Java), the host language compiler is n…

> lambdas Lambdas in the host language won't play well with remote RDBMSes -- you'd have to be able to serialize the lambda and make the serialized form reasonably efficient. I'm skeptical of lambdas for expressions in DB query APIs. An expression like `foo + bar` has to become `.expr(plus("foo", "bar"))`, except, if you take this to its limit you'll want to use non-string objects to identify column names and other s…

"And yet an API is kinda desirable."

Yeah. however I may not live to see the day when that desire, which I agree is felt by 99.99% of the developer community (hell, even by 99.99% of the end user community because don't come and tell me that what that community is feeling isn't some sort of sense that "the developers just can't offer us any answers"), actually gets to be fulfilled.

"compiling to a standard AST"

I think that what you might be failing to appreciate is that achieving that requires a "standard algebra", and that no such thing exists at this present day.

Post reply on HN