Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wrote a SQL engine in Python

github.com

Post reply on HN
Print view View original

61–70 of 81 posts

Re: I wrote a SQL engine in Python

#61

Post by int3 »

I don't have very much background in ML or distributed systems, so forgive my naive questions...

> After all, most ML in industry today seems to be lightweight models applied to heavily engineered features

I assume "lightweight models" are those that don't have too many parameters, and "heavily engineered features" mean that the data fed into the model has undergone significant pre-processing via potentially complicated UDFs -- hence the motivation for the project. Is that right?

> Quokka is an open-source push-based vectorized query engine ... it is meant to be much more performant than blocking-shuffle based alternatives like SparkSQL

Does anyone have pointers to what push-based vs blocking-shuffle engines are? Any good papers?

> It should work on local machine no problem (and should be a lot faster than Pandas!)

So I understand why Quokka is faster than Spark, but I'm a bit uncertain as to why the author is also making a comparison with Pandas on a single machine. Is it because the streaming pipeline design means that Quokka can better take advantage of multiple cores?

Re: I wrote a SQL engine in Python

#62

Post by marsupialtail_2 »

int3 wrote: post #61

I don't have very much background in ML or distributed systems, so forgive my naive questions... > After all, most ML in industry today seems to be lightweight models applied to heavily engineered features I assume "lightweight models" are those that don't have too many parameters, and "heavily engineered features" mean that the data fed into the model has undergone significant pre-processing via potentially complica…

That's right. My background is mostly in quantitative finance, where we would use models like linear regression on expert-engineered features based on market data, instead of throwing a deep neural network at raw price data like what some people might imagine.

For push vs. pull, I'd recommend: https://news.ycombinator.com/item?id=27006476.

On single machine, you really should just use Polars. Quokka is faster than Pandas because it can take advantage of multiple cores, but so can Polars -- and it is likely to be faster.

Re: I wrote a SQL engine in Python

#67

Post by throwaway290 »

marsupialtail_2 wrote: post #65
throwaway290 wrote: post #64

> Having lost all the money I made from my startup on shitcoins and the stock market, I returned to my PhD program to build a better distributed query engine, Quokka Casual README slip of the year...

Let's make this top-rated comment :-P

I will be the first to admit I didn't add much of value, so if it actually happens I will have to downgrade my opinion of HN audience

Re: I wrote a SQL engine in Python

#68

Post by int3 »

marsupialtail_2 wrote: post #62
int3 wrote: post #61

I don't have very much background in ML or distributed systems, so forgive my naive questions... > After all, most ML in industry today seems to be lightweight models applied to heavily engineered features I assume "lightweight models" are those that don't have too many parameters, and "heavily engineered features" mean that the data fed into the model has undergone significant pre-processing via potentially complica…

That's right. My background is mostly in quantitative finance, where we would use models like linear regression on expert-engineered features based on market data, instead of throwing a deep neural network at raw price data like what some people might imagine. For push vs. pull, I'd recommend: https://news.ycombinator.com/item?id=27006476 . On single machine, you really should just use Polars. Quokka is faster than P…

Thanks for the answers! And the push vs pull link is a great explanation indeed :)

Re: I wrote a SQL engine in Python

#69

Post by _zoltan_ »

tmtvl wrote: post #50
marsupialtail_2 wrote: post #41

Earlier quoted context omitted.

I only wrote Python. I am using C++ and Rust libraries

If I write a shell script looking something like: #!/bin/sh exec rg $@ Then I did not actually write ripgrep in pure shell.

your example clearly has nothing to do with the OP's claims.

Re: I wrote a SQL engine in Python

#70

Post by dvko »

akdor1154 wrote: post #43

I'm interested in this just for the use case of 'a sql frontend for polars' - I wonder if just that part could be used independently?

Polars has basic SQL support already, it’s just not that well documented. You’ll need to compile Polars with the ‘sql’ feature flag.

Not sure what the process looks like through the Python API. Maybe @ritchie46 can chime in?

Post reply on HN