41–50 of 81 posts
Earlier quoted context omitted.
that's certainly one way to look at it
Am I incorrect? Elsewhere in the thread you say this project is pure Python and then in the post you say you use a Rust library for queries. Am I misunderstanding something?
I wrote a toy distributed SQL, cypher graph, dynamodb style and document storage Python database but it's more for experimentation than serious use. It's not ready for use it's more a show of how little code you can use to write a database. https://GitHub.com/samsquire/hash-db
Something like building a toy dynamodb variant. Would pay good money for this.
Is one thing to read about and guesstimate implementation choices from white paper. Totally another if databases/distributed systems expert walks you through propagating write ahead logs to replicas. Hehe
And what it means to have eventual consistent writes vs strong writes in practice. A way to teach by doing.
It might be worth running your benchmarks against Trino with fault tolerant execution mode enabled. Check the documentation here: https://trino.io/docs/current/admin/fault-tolerant-execution...
Adding fault tolerant to execution to Trino was a big and complicated project for anyone interested in more details check here: https://trino.io/blog/2022/05/05/tardigrade-launch.html
Earlier quoted context omitted.
Am I incorrect? Elsewhere in the thread you say this project is pure Python and then in the post you say you use a Rust library for queries. Am I misunderstanding something?
Can't a pure Python project use any library that's not a pure Python project? Is python-non-purity something that taints all dependants for you?
For example, here [0] is a "pure Python postgres driver" and the implication is that it doesn't use libpg.
Or see also this discussion [1].
I don't mean that the author should or shouldn't do whatever they want. I just wouldn't use the term "pure" here or even say this is an engine written in Python since a large chunk of it is not written in Python. That doesn't mean this is not a cool project. It's a cool project!
[0] https://github.com/tlocke/pg8000
[1] https://www.reddit.com/r/learnpython/comments/nktut1/eli5_th...
Trino can be fault tolerant but you have to explicitly enable fault tolerant execution. It might be worth running your benchmarks against Trino with fault tolerant execution mode enabled. Check the documentation here: https://trino.io/docs/current/admin/fault-tolerant-execution... Adding fault tolerant to execution to Trino was a big and complicated project for anyone interested in more details check here: https://tr…
I'm interested in this just for the use case of 'a sql frontend for polars' - I wonder if just that part could be used independently?
~One suggestion, a Scala/Java Spark run of those benchmarks should be a valid baseline to compare against as well instead of PySpark.~ Ah it's SparkSQL so the execution probably wouldn't have much of py4j involvement, except for the collect.
One significant disadvantage of PySpark is its reliance on py4j to serialize and deserialize objects between Java and Python when using Python UDFs. This constant overhead can become burdensome as data volume increases in such an exchange. However, I am glad to see efforts to create a data pipeline framework using Python and Ray. ~One suggestion, a Scala/Java Spark run of those benchmarks should be a valid baseline t…
Earlier quoted context omitted.
Am I incorrect? Elsewhere in the thread you say this project is pure Python and then in the post you say you use a Rust library for queries. Am I misunderstanding something?
I only wrote Python. I am using C++ and Rust libraries
#!/bin/sh
exec rg $@