I wrote a SQL engine in Python
github.com
I wrote a SQL engine in Python
1–10 of 81 posts
Re: I wrote a SQL engine in Python
#2I think the most interesting part of this project is the fault tolerance. I can’t say I’ve seen any other projects do this, but it seems reasonable to want checkpointing during a long computation.
Another thing I like is that conceptually it seems like it would be simple to switch the underlying query engine (right now it’s Polars) in the future. Seems like a pretty general distributed system.
Re: I wrote a SQL engine in Python
#3I think the most interesting part of this project is the fault tolerance. I can’t say I’ve seen any other projects do this, but it seems reasonable to want checkpointing during a long computation. Another thing I like is that conceptually it seems like it would be simple to switch the underlying query engine (right now it’s Polars) in the future. Seems like a pretty general distributed system.
Perhaps you can write the underlying query engine :-) in something that's not Python lol
Re: I wrote a SQL engine in Python
#4I think the most interesting part of this project is the fault tolerance. I can’t say I’ve seen any other projects do this, but it seems reasonable to want checkpointing during a long computation. Another thing I like is that conceptually it seems like it would be simple to switch the underlying query engine (right now it’s Polars) in the future. Seems like a pretty general distributed system.
Checkpointing is actually a bit difficult for fault tolerance for long running batch computations and state tends to grow linearly as the computation progresses, unlike streaming applications where you typically use windows. This means periodic checkpointing leads to quadratic amount of writes to durable storage which is pretty bad.
Quokka supports checkpointing but does not enable that by default to prevent this common problem from killing normal operation performance.
Re: I wrote a SQL engine in Python
#5Can you explain how this might differ from something like https://github.com/apache/arrow-ballista
I've seen several variants of "next-gen" spark, but nowhere have I really seen the different tradeoffs/advantages/disadvantages between them.
Re: I wrote a SQL engine in Python
#6Can you explain how this might differ from something like https://github.com/apache/arrow-ballista I've seen several variants of "next-gen" spark, but nowhere have I really seen the different tradeoffs/advantages/disadvantages between them.
Firstly quokka is pure Python, which I believe is good for interoperability with Python based UDFs.
Secondly quokka tries to be fault tolerant. I.e. it can handle worker failures intra query and not have to start over. This is quite important in real world spark deployments with thousands of nodes running many hours. AFAIK this is not well supported by most spark alternatives.
Finally quokka has a much stronger focus on time series data analytics. It is meant to excel at workloads like range joins and asof/PIT joins used for feature engineering. (This part isn't too stable yet so is not open source)
This means quokka optimizes on a different point in the UDF/performance/fault tolerance tradeoff space than something like arrow ballista or starrocks, which I think are pure performance plays
Re: I wrote a SQL engine in Python
#7Can you explain how this might differ from something like https://github.com/apache/arrow-ballista I've seen several variants of "next-gen" spark, but nowhere have I really seen the different tradeoffs/advantages/disadvantages between them.
Firstly quokka is pure Python, which I believe is good for interoperability with Python based UDFs. Secondly quokka tries to be fault tolerant. I.e. it can handle worker failures intra query and not have to start over. This is quite important in real world spark deployments with thousands of nodes running many hours. AFAIK this is not well supported by most spark alternatives. Finally quokka has a much stronger focus…
Interesting, I was wondering if you considered building on top of https://github.com/apache/arrow-datafusion-python
I really do think a distributed db with compute/storage separation and optimized for feature engineering/dataloading (for training NNs) is underserved.
I'd be very interested in the time series aspects of what you're building.
Re: I wrote a SQL engine in Python
#8Earlier quoted context omitted.
Firstly quokka is pure Python, which I believe is good for interoperability with Python based UDFs. Secondly quokka tries to be fault tolerant. I.e. it can handle worker failures intra query and not have to start over. This is quite important in real world spark deployments with thousands of nodes running many hours. AFAIK this is not well supported by most spark alternatives. Finally quokka has a much stronger focus…
Interesting, I was wondering if you considered building on top of https://github.com/apache/arrow-datafusion-python I really do think a distributed db with compute/storage separation and optimized for feature engineering/dataloading (for training NNs) is underserved. I'd be very interested in the time series aspects of what you're building.
hmm I wasn't aware of https://github.com/apache/arrow-datafusion-python... thanks for the pointer.
time series target release by April this year. main challenge is supporting them in the SQL API -- execution engine support is already done
Re: I wrote a SQL engine in Python
#9Python interpreter in SQL - the reverse of what was done - would've been really impressive. Terrible idea of course but impressive nonetheless.
Re: I wrote a SQL engine in Python
#10Python interpreter in SQL - the reverse of what was done - would've been really impressive. Terrible idea of course but impressive nonetheless.
A Python to SQL compiler already exists https://db.cs.uni-tuebingen.de/publications/2022/snakes-on-a...