I like good sql projects like this. I use duckdb for almost everything.
I wrote a SQL engine in Python
71–80 of 81 posts
Re: I wrote a SQL engine in Python
#72I'm interested in this just for the use case of 'a sql frontend for polars' - I wonder if just that part could be used independently?
Polars has basic SQL support already, it’s just not that well documented. You’ll need to compile Polars with the ‘sql’ feature flag. Not sure what the process looks like through the Python API. Maybe @ritchie46 can chime in?
Yes, we have basic support.
Here are some examples of how to use it in python:
https://github.com/pola-rs/polars/blob/91a419acaf024e64410e7...
However, full sql support is on the roadmap. It's just a matter of hours in a day...
Re: I wrote a SQL engine in Python
#73> A library to parse and optimize SQL, That's like saying "a library to parse and optimize computer programs", except probably even harder, since a compiler and runtime library can't make any assumptions about the programs they need to make run, so they're limited in the potential of utilizing all that context information. Countless person-years have been spent on this and it's still a very active fields of research…
I'm the author of SQLGlot. Optimizing SQL is quite common and means things like projection / predicate push downs and logical simplification. SQL can be optimized relatively easily compared to a programming language because it is declarative. For example SELECT * FROM (SELECT * FROM x) WHERE z = 1 can be optimized into SELECT * FROM x where z = 1
Oh, you mean optimizing the SQL as SQL, not optimizing the query execution?
Well, some of these transformations are useful, like the one you presented. But any non-trivial transformation may be either beneficial or detrimental, depending on a myriad of factors including memory layout, compression, distribution of data, computing hardware etc.
Re: I wrote a SQL engine in Python
#74Earlier quoted context omitted.
Am I incorrect? Elsewhere in the thread you say this project is pure Python and then in the post you say you use a Rust library for queries. Am I misunderstanding something?
I only wrote Python. I am using C++ and Rust libraries
"Pure Python" ultimately means that it all runs within the python interpreter, with no dependencies on code that runs outside of the interpreter (standard library notwithstanding).
Re: I wrote a SQL engine in Python
#75I have previously used Dask to handle larger data sets, and that made me wonder - how does Quokka compare to Dask?
Re: I wrote a SQL engine in Python
#76I think the most interesting part of this project is the fault tolerance. I can’t say I’ve seen any other projects do this, but it seems reasonable to want checkpointing during a long computation. Another thing I like is that conceptually it seems like it would be simple to switch the underlying query engine (right now it’s Polars) in the future. Seems like a pretty general distributed system.
Perhaps you can write the underlying query engine :-) in something that's not Python lol
Polars is already not python
Re: I wrote a SQL engine in Python
#77The core of it is Rust: > Very fast kernels for SQL primitives like joins, filtering and aggregations. Quokka uses Polars to implement these. (I sponsor Polars on Github and you should too.) I am also exploring DuckDB, but I have found Polars to be faster so far.
PyArrow is also a CPython extension wrapping a C++ library. I agree that it’s a little disingenuous to call it “pure Python” when the two libraries doing the heavy lifting are non-Python; but it’s not a lie that the entirety of the Quokka-specific codebase is Python. Personally, what I would be more interested in (and what I thought this would be from the title) is a full SQL engine wholesale coded in Python, a la SQ…
> Personally, what I would be more interested in (and what I thought this would be from the title) is a full SQL engine wholesale coded in Python, a la SQLite. Even if it wasn’t super performant or functional.
Here you go!
Re: I wrote a SQL engine in Python
#78Earlier quoted context omitted.
Even the self driving in Tesla use Python. If your Python is slow, you are using Python wrong. So it’s kind of a good thing that it’s natively slow, because you are not supposed to do your computations with native Python. It it was faster (added a lot of type hints and build processes), you would probably not be nudged in the right direction. https://cerfacs.fr/coop/fortran-vs-python
Self driving for TSLA cars is not really the benchmark I'd use for "something that works reliably"
I don’t think there’s any evidence that FSD’s reliability is affected by being written in Python.
Re: I wrote a SQL engine in Python
#79Thanks for sharing.
I have a SQL Engine in Python too (https://github.com/mabel-dev/opteryx). I focused my initial effort on supporting SQL statements and making the usage feel like a database - that probably reflects the problem I had in front of me when I set out - only handling handfuls of gigabytes in a batch environment for ETLs with a group of new-to-data-engineering engineers. Have recently started looking more at real-time performance, such as distributing work. Am interesting in how you've approached.
Re: I wrote a SQL engine in Python
#80Thanks for sharing. I have a SQL Engine in Python too ( https://github.com/mabel-dev/opteryx ). I focused my initial effort on supporting SQL statements and making the usage feel like a database - that probably reflects the problem I had in front of me when I set out - only handling handfuls of gigabytes in a batch environment for ETLs with a group of new-to-data-engineering engineers. Have recently started looking m…
Wait this is really cool. How do you parse and optimize SQL?