Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I wrote a SQL engine in Python

github.com

Post reply on HN
Print view View original

51–60 of 81 posts

Re: I wrote a SQL engine in Python

#51

Post by Epa095 »

nivekkevin wrote: post #48

One significant disadvantage of PySpark is its reliance on py4j to serialize and deserialize objects between Java and Python when using Python UDFs. This constant overhead can become burdensome as data volume increases in such an exchange. However, I am glad to see efforts to create a data pipeline framework using Python and Ray. ~One suggestion, a Scala/Java Spark run of those benchmarks should be a valid baseline t…

There is also pandas udfs, which uses arrow as the exchange format. I assume it still has to copy the data (?), but it makes the (de)serializarion fast, and allows for vectorized operations.

https://spark.apache.org/docs/3.0.0/sql-pyspark-pandas-with-...

Re: I wrote a SQL engine in Python

#52

Post by samsquire »

birdymcbird wrote: post #42
samsquire wrote: post #37

I wrote a toy distributed SQL, cypher graph, dynamodb style and document storage Python database but it's more for experimentation than serious use. It's not ready for use it's more a show of how little code you can use to write a database. https://GitHub.com/samsquire/hash-db

Always wanted to have something like the crafting interpreters style book but implementing a database from scratch, and bonus points for a distributed system with a leader/replica model, leader election and things like partitioning, resizing partitions, handling node failure. Something like building a toy dynamodb variant. Would pay good money for this. Is one thing to read about and guesstimate implementation choice…

I ought to spend some time documenting the code.

I prefer reading a description of an algorithm than the code of the algorithm. From the description I can work out how to implement the code myself. I might look at someone's code for ideas or compare how I solved a problem.

I too would enjoy a whitepaper of database design. I document all whitepapers read on GitHub profile (see my HN profile)

I have a simple btree here https://GitHub.com/samsquire/btree

Farley Knight added an AVL tree to the repository that I need to change the unbalanced tree to use it.

I would like to know how SQLite VM works so I can write a database VM. I learned some things from crafting interpreters book and started writing my own programming language here

https://GitHub.com/samsquire/multiversion-concurrency-contro...

Re: I wrote a SQL engine in Python

#53

Post by app4soft »

marsupialtail_2 wrote: post #30
totalhack wrote: post #25

Sorry if I missed it -- Are there plans to offer a way to query this in actual sql? I believe SingleStore is MySQL compatible for example which I think is a nice feature. Basically I want to be able to interact with this much like I'd interact with another database I'm using or perhaps with a sqlalchemy core integration (which both SingleStore and Snowflake have).

Yes -- we are building a sql compiler to the dataframe API. It currently passes half of TPC-H, but is not really ready. It will be open-sourced soon. The SQL optimizations like predicate pushdown and early projection are all there already in the dataframe API, similar to Polars.

> It will be open-sourced soon.

Just reminds many of such projects like Lightworks:

> Editor's note: The intent for Lightworks to go open source seems to have been abandoned.

[0] https://opensource.com/business/12/10/lightworks-linux-devel...

Re: I wrote a SQL engine in Python

#54

Post by [deleted] »

samsquire wrote: post #37

I wrote a toy distributed SQL, cypher graph, dynamodb style and document storage Python database but it's more for experimentation than serious use. It's not ready for use it's more a show of how little code you can use to write a database. https://GitHub.com/samsquire/hash-db

[deleted]

Re: I wrote a SQL engine in Python

#55

Post by deaddodo »

eatonphil wrote: post #29

The core of it is Rust: > Very fast kernels for SQL primitives like joins, filtering and aggregations. Quokka uses Polars to implement these. (I sponsor Polars on Github and you should too.) I am also exploring DuckDB, but I have found Polars to be faster so far.

PyArrow is also a CPython extension wrapping a C++ library.

I agree that it’s a little disingenuous to call it “pure Python” when the two libraries doing the heavy lifting are non-Python; but it’s not a lie that the entirety of the Quokka-specific codebase is Python.

Personally, what I would be more interested in (and what I thought this would be from the title) is a full SQL engine wholesale coded in Python, a la SQLite. Even if it wasn’t super performant or functional.

Re: I wrote a SQL engine in Python

#56

Post by asadawadia »

captaintobs wrote: post #23
asadawadia wrote: post #22

Earlier quoted context omitted.

sql can be arbitrarily nested/deep - so how does the code 'know' what to do

recursion! https://github.com/tobymao/sqlglot/tree/main/sqlglot/optimiz...

Does this only optimize things like `select * from table where false` type stuff or can it optimize access paths as well? such as whether an index can be used? and what ranges? or should a sequential scan be used

Re: I wrote a SQL engine in Python

#58

Post by captaintobs »

asadawadia wrote: post #56
captaintobs wrote: post #23

Earlier quoted context omitted.

recursion! https://github.com/tobymao/sqlglot/tree/main/sqlglot/optimiz...

Does this only optimize things like `select * from table where false` type stuff or can it optimize access paths as well? such as whether an index can be used? and what ranges? or should a sequential scan be used

it can do everything that can be done with the query and schema. it doesn't yet do indices or physical statistics

Re: I wrote a SQL engine in Python

#59

Post by cpard »

marsupialtail_2 wrote: post #46
cpard wrote: post #44

Trino can be fault tolerant but you have to explicitly enable fault tolerant execution. It might be worth running your benchmarks against Trino with fault tolerant execution mode enabled. Check the documentation here: https://trino.io/docs/current/admin/fault-tolerant-execution... Adding fault tolerant to execution to Trino was a big and complicated project for anyone interested in more details check here: https://tr…

I have. I am about 2x faster than trino with fault tolerance. But I didn't put the numbers on that plot because this trino feature is still really new and I might not be benchmarking it in the best way.

That’s awesome. For anything Trino - project Tardigrade related, reach out to any of the maintainers, they’ll be happy to help.
Post reply on HN