Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wrote a SQL engine in Python

github.com

Post reply on HN
Print view View original

21–30 of 81 posts

Re: I wrote a SQL engine in Python

#21

Post by punnerud »

marsupialtail_2 wrote: post #3
sakras wrote: post #2

I think the most interesting part of this project is the fault tolerance. I can’t say I’ve seen any other projects do this, but it seems reasonable to want checkpointing during a long computation. Another thing I like is that conceptually it seems like it would be simple to switch the underlying query engine (right now it’s Polars) in the future. Seems like a pretty general distributed system.

Perhaps you can write the underlying query engine :-) in something that's not Python lol

Even the self driving in Tesla use Python. If your Python is slow, you are using Python wrong. So it’s kind of a good thing that it’s natively slow, because you are not supposed to do your computations with native Python. It it was faster (added a lot of type hints and build processes), you would probably not be nudged in the right direction.

https://cerfacs.fr/coop/fortran-vs-python

Re: I wrote a SQL engine in Python

#22

Post by asadawadia »

captaintobs wrote: post #17
einpoklum wrote: post #12

> A library to parse and optimize SQL, That's like saying "a library to parse and optimize computer programs", except probably even harder, since a compiler and runtime library can't make any assumptions about the programs they need to make run, so they're limited in the potential of utilizing all that context information. Countless person-years have been spent on this and it's still a very active fields of research…

I'm the author of SQLGlot. Optimizing SQL is quite common and means things like projection / predicate push downs and logical simplification. SQL can be optimized relatively easily compared to a programming language because it is declarative. For example SELECT * FROM (SELECT * FROM x) WHERE z = 1 can be optimized into SELECT * FROM x where z = 1

sql can be arbitrarily nested/deep - so how does the code 'know' what to do

Re: I wrote a SQL engine in Python

#23

Post by captaintobs »

asadawadia wrote: post #22
captaintobs wrote: post #17

Earlier quoted context omitted.

I'm the author of SQLGlot. Optimizing SQL is quite common and means things like projection / predicate push downs and logical simplification. SQL can be optimized relatively easily compared to a programming language because it is declarative. For example SELECT * FROM (SELECT * FROM x) WHERE z = 1 can be optimized into SELECT * FROM x where z = 1

sql can be arbitrarily nested/deep - so how does the code 'know' what to do

recursion!

https://github.com/tobymao/sqlglot/tree/main/sqlglot/optimiz...

Re: I wrote a SQL engine in Python

#24

Post by divbzero »

sakras wrote: post #2

I think the most interesting part of this project is the fault tolerance. I can’t say I’ve seen any other projects do this, but it seems reasonable to want checkpointing during a long computation. Another thing I like is that conceptually it seems like it would be simple to switch the underlying query engine (right now it’s Polars) in the future. Seems like a pretty general distributed system.

Fault tolerance is an important feature but, given Python’s popularity in data science, the most interesting part of this project to me is support for Python UDFs.

In principle, the programming language should not be the greatest consideration because developers can learn and use different languages for different applications. In practice, being able to draw on familiar syntax and libraries can make a real difference in usability.

Re: I wrote a SQL engine in Python

#25

Post by totalhack »

Sorry if I missed it -- Are there plans to offer a way to query this in actual sql? I believe SingleStore is MySQL compatible for example which I think is a nice feature. Basically I want to be able to interact with this much like I'd interact with another database I'm using or perhaps with a sqlalchemy core integration (which both SingleStore and Snowflake have).

Re: I wrote a SQL engine in Python

#27

Post by PontifexMinimus »

KMag wrote: post #20
jkeddo wrote: post #16

Earlier quoted context omitted.

Why?

Most likely an autodidactic exercise. I'd love to see the source, even if, or especially if, it's half-finished. Seeing someone work their way through the snarls in such a project would be useful.

Partly that.

Re: I wrote a SQL engine in Python

#28

Post by imiric »

I haven't looked into this in detail, and it seems like a fine project at a glance, but this caught my attention from the introduction:

> When I set out, I had several objectives:

> Easy to install and run, especially for distributed deployments.

> [...]

> The first two objectives strongly scream Python as the language of choice for Quokka.

Python is probably one of the last languages I'd consider if ease of deployment is a priority. Packaging has historically been a mess, and deploying standalone binaries across platforms is a pain. State of the art solutions are 3rd party and involve bundling the intepreter for each platform. It's been a few years since I last used it for anything serious, but I believe this is still the case.

Whereas something like Go actually makes this infinitely easier, for both the developer and the user. One native Go command builds a standalone binary for each platform. It couldn't be simpler.

The other objective of supporting Python UDFs necessarily ties you to Python. And since this is solving a data science problem, it makes sense for it to be written in Python.

Re: I wrote a SQL engine in Python

#29

Post by eatonphil »

The core of it is Rust:

> Very fast kernels for SQL primitives like joins, filtering and aggregations. Quokka uses Polars to implement these. (I sponsor Polars on Github and you should too.) I am also exploring DuckDB, but I have found Polars to be faster so far.

Re: I wrote a SQL engine in Python

#30

Post by marsupialtail_2 »

totalhack wrote: post #25

Sorry if I missed it -- Are there plans to offer a way to query this in actual sql? I believe SingleStore is MySQL compatible for example which I think is a nice feature. Basically I want to be able to interact with this much like I'd interact with another database I'm using or perhaps with a sqlalchemy core integration (which both SingleStore and Snowflake have).

Yes -- we are building a sql compiler to the dataframe API. It currently passes half of TPC-H, but is not really ready. It will be open-sourced soon.

The SQL optimizations like predicate pushdown and early projection are all there already in the dataframe API, similar to Polars.

Post reply on HN