I wrote a SQL engine in Python • Hacker News Board

Re: I wrote a SQL engine in Python

#11

Post by marsupialtail_2 » Fri, Dec 30, 2022, 10:01 PM UTC

hbirler wrote: post #10

brunobowden wrote: post #9

Python interpreter in SQL - the reverse of what was done - would've been really impressive. Terrible idea of course but impressive nonetheless.

A Python to SQL compiler already exists https://db.cs.uni-tuebingen.de/publications/2022/snakes-on-a...

I don't really compile python functions to sql -- I just execute sql. It's just that the execution engine is written in python

Re: I wrote a SQL engine in Python

#12

Post by einpoklum » Fri, Dec 30, 2022, 10:04 PM UTC

> A library to parse and optimize SQL,

That's like saying "a library to parse and optimize computer programs", except probably even harder, since a compiler and runtime library can't make any assumptions about the programs they need to make run, so they're limited in the potential of utilizing all that context information.

Countless person-years have been spent on this and it's still a very active fields of research and engineering.

> 2x SparkSQL performance

Ah, ok, so it can be slow. Never mind then, carry on :-P

Re: I wrote a SQL engine in Python

#13

Post by marsupialtail_2 » Fri, Dec 30, 2022, 10:07 PM UTC

einpoklum wrote: post #12

> A library to parse and optimize SQL, That's like saying "a library to parse and optimize computer programs", except probably even harder, since a compiler and runtime library can't make any assumptions about the programs they need to make run, so they're limited in the potential of utilizing all that context information. Countless person-years have been spent on this and it's still a very active fields of research…

SQLGlot is really great, check it out!

Re: I wrote a SQL engine in Python

#14

Post by zitterbewegung » Fri, Dec 30, 2022, 10:16 PM UTC

Have you tested this with Jepsen?

Re: I wrote a SQL engine in Python

#15

Post by PontifexMinimus » Fri, Dec 30, 2022, 10:25 PM UTC

Funnily enough I'm currently writing a NoSQL database in Python.

Re: I wrote a SQL engine in Python

#16

Post by jkeddo » Fri, Dec 30, 2022, 10:28 PM UTC

PontifexMinimus wrote: post #15

Funnily enough I'm currently writing a NoSQL database in Python.

Why?

Re: I wrote a SQL engine in Python

#17

Post by captaintobs » Fri, Dec 30, 2022, 10:28 PM UTC

einpoklum wrote: post #12

> A library to parse and optimize SQL, That's like saying "a library to parse and optimize computer programs", except probably even harder, since a compiler and runtime library can't make any assumptions about the programs they need to make run, so they're limited in the potential of utilizing all that context information. Countless person-years have been spent on this and it's still a very active fields of research…

I'm the author of SQLGlot. Optimizing SQL is quite common and means things like projection / predicate push downs and logical simplification. SQL can be optimized relatively easily compared to a programming language because it is declarative. For example

SELECT * FROM (SELECT * FROM x) WHERE z = 1

can be optimized into

SELECT * FROM x where z = 1

Re: I wrote a SQL engine in Python

#18

Post by marsupialtail_2 » Fri, Dec 30, 2022, 10:32 PM UTC

zitterbewegung wrote: post #14

Have you tested this with Jepsen?

The focus is olap and etl not oltp. In fact I assume data doesn't change :-)

Re: I wrote a SQL engine in Python

#19

Post by marsupialtail_2 » Fri, Dec 30, 2022, 10:33 PM UTC

PontifexMinimus wrote: post #15

Funnily enough I'm currently writing a NoSQL database in Python.

Name?

Re: I wrote a SQL engine in Python

#20

Post by KMag » Fri, Dec 30, 2022, 10:43 PM UTC

jkeddo wrote: post #16

PontifexMinimus wrote: post #15

Funnily enough I'm currently writing a NoSQL database in Python.

Why?

Most likely an autodidactic exercise. I'd love to see the source, even if, or especially if, it's half-finished. Seeing someone work their way through the snarls in such a project would be useful.