Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python

github.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python

#61

Post by Eddygandr »

crabbone wrote: post #58
samsquire wrote: post #5

Thank you for sharing My perspective is that writing this kind of system in a language such as Python is actually a great thing because for myself Python is more widely readable and approachable compared to C++ or C which is what databases are often programmed in. If someone wants to be serious they can port it to a low level language . As it stands it's educational and useful for studying. I wrote a distributed pseu…

Idk. Python is just as bad as C / C++, with the downside that you cannot do much of the interesting stuff one would need to if they wanted to learn how to make databases, if they use Python. Both C and Python are very "approachable" if you ignore the bad language design, inconsistencies and other "gotchas" and only take the "easy" parts, disregarding edge cases. However, with C you could at least have a fighting chan…

> but with Python you'll never even know what the real thing is like

What do you mean? Surely the concepts are similar regardless of using Python or C?

Re: Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python

#62

Post by dumpsterdiver »

Eddygandr wrote: post #61
crabbone wrote: post #58

Earlier quoted context omitted.

Idk. Python is just as bad as C / C++, with the downside that you cannot do much of the interesting stuff one would need to if they wanted to learn how to make databases, if they use Python. Both C and Python are very "approachable" if you ignore the bad language design, inconsistencies and other "gotchas" and only take the "easy" parts, disregarding edge cases. However, with C you could at least have a fighting chan…

> but with Python you'll never even know what the real thing is like What do you mean? Surely the concepts are similar regardless of using Python or C?

The concepts and syntax are similar, but Python will always be "riding upon a horse", whereas C can be the horse.

Re: Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python

#64

Post by mejutoco »

OJFord wrote: post #36
tmpX7dMeXU wrote: post #26

Earlier quoted context omitted.

It’s an existing, standard, language for describing grammars. Quite clearly the trade-off being made here. Seemingly not the one you would’ve made. Don’t act like it’s objectively bad.

in a string . I didn't say the DSL was novel. I learnt BNF too. (It does have extras that are either unique to it or standard beyond my familiarity though.)

I was always surprised at how the Ruby community talks about DSLs when using Ruby classes and catch all methods, overloading, etc. Nothing bad with it, but I feel it is not really a language, just very dynamic Ruby code.

A DSL that is a string, while having some downsides, does not have the arbitrary limitations of the "host" language. Either approach has its pros an cons.

Re: Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python

#65

Post by crabbone »

Eddygandr wrote: post #61
crabbone wrote: post #58

Earlier quoted context omitted.

Idk. Python is just as bad as C / C++, with the downside that you cannot do much of the interesting stuff one would need to if they wanted to learn how to make databases, if they use Python. Both C and Python are very "approachable" if you ignore the bad language design, inconsistencies and other "gotchas" and only take the "easy" parts, disregarding edge cases. However, with C you could at least have a fighting chan…

> but with Python you'll never even know what the real thing is like What do you mean? Surely the concepts are similar regardless of using Python or C?

Not even close.

The real problems you will have to solve when working with a database anyone would want to use are, for example:

* Memory layout of the buffers used to write / read / cache this data. In Python, you don't even have a concept of memory alignment / layout -- it's all happening somewhere in the interpreter.

* How to best service multiple requests concurrently. Again, Python offers nothing here, and nothing to experiment with. But this is a huge part of working with databases. The whole two-stage commit, transaction, consistency guarantees -- it's the central point of databases, but Python gives you no tools to even try anything like that.

* Working with various underlying storage... most of it has no Python interface (but does have C interface). Eg. if you want to understand how to optimize storing data by using some filesystem -- those filesystems do often expose similar interfaces that go beyond VFS, but they won't be immediately available to Python.

* Working with vectorization of queries -- again, Python doesn't have a concept of ISA, doesn't have any way to instruct the code to utilize any particular CPU instructions... but this is where a lot of work is done by people who work on real databases. Not being able to get any meaningful access to query optimization, planning becomes pointless / useless -- what are your concerns going to be when you write a query planner if you still have no idea how it's going to be executed?

* Similar stuff goes for networking -- whenever you need to solve something that goes beyond the absolute trivial you will at best rely on Python bindings to some library that actually does that rather than on Python code proper. In other words, if your goal was to learn how to do it, you will not achieve that goal because the actual work will happen elsewhere.

So... I don't know... there is no way you can really learn how to make databases in Python. You can probably learn something, depending on what's your baseline, but you will not be ready to make a real thing, if all you have is Python. It's a difference between toy doctor set and learning to be a surgeon...

Re: Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python

#66

Post by FrustratedMonky »

crabbone wrote: post #65
Eddygandr wrote: post #61

Earlier quoted context omitted.

> but with Python you'll never even know what the real thing is like What do you mean? Surely the concepts are similar regardless of using Python or C?

Not even close. The real problems you will have to solve when working with a database anyone would want to use are, for example: * Memory layout of the buffers used to write / read / cache this data. In Python, you don't even have a concept of memory alignment / layout -- it's all happening somewhere in the interpreter. * How to best service multiple requests concurrently. Again, Python offers nothing here, and nothi…

Guess to get back to the post. He did it for learning, not to be an exact duplicate or competitor to replace other RDBMS.

Often when learning, you do not implement every difficult edge case, the most complex. You are just trying to get the jist of it. You want a smaller problem to solve.

Maybe as a learning project, it isn't important to have concurrency, networking, memory management. Unless, any of those things happens to be of interest to learn about also, then add them back in.

I don't think this is trying to be an argument for Python as good to build an RDB in. (of course it isn't for all the reasons you list)

Python just happens to be an easy language for beginners, so why not build a basic RDB to learn about that too.

Re: Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python

#67

Post by lopatin »

devman0 wrote: post #8
samsquire wrote: post #5

Thank you for sharing My perspective is that writing this kind of system in a language such as Python is actually a great thing because for myself Python is more widely readable and approachable compared to C++ or C which is what databases are often programmed in. If someone wants to be serious they can port it to a low level language . As it stands it's educational and useful for studying. I wrote a distributed pseu…

This is why you see a community of pure Java RDBMSs (Hypersonic, H2, Derby, etc), if you don't need the big iron scale it's easier to ship/use the database or even embed it in memory if needed.

H2 has quite heavy use in production too. For example Apache Ignite uses it as the core of it's distributed SQL engine. Ignite takes a distributed SQL query and splits it up into single node queries, which are executed on each node by H2, and then Ignite merges the results back.

Re: Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python

#68

Post by flakiness »

This is very nice!

SQLite is very hard to read, but this one is actually quite comprehensible. Especially the VM part: https://github.com/spandanb/learndb-py/blob/master/learndb/v...

Compare it with this: https://github.com/sqlite/sqlite/blob/master/src/vdbe.c

That's said, I'm curious how complete this LearnDB is. SQLite is hard to read not only it's old but also it covers a lot of SQL and following SQL spec makes hings complicated. SQLite has great test suite so it's nice if you run the suit against this implementation.

Re: Show HN: I wrote a RDBMS (SQLite clone) from scratch in pure Python

#69

Post by OJFord »

mejutoco wrote: post #64
OJFord wrote: post #36

Earlier quoted context omitted.

in a string . I didn't say the DSL was novel. I learnt BNF too. (It does have extras that are either unique to it or standard beyond my familiarity though.)

I was always surprised at how the Ruby community talks about DSLs when using Ruby classes and catch all methods, overloading, etc. Nothing bad with it, but I feel it is not really a language, just very dynamic Ruby code. A DSL that is a string, while having some downsides, does not have the arbitrary limitations of the "host" language. Either approach has its pros an cons.

I'm not that versed in Ruby, but I think similarly say the Django ORM in Python borders on DSL - especially with use of bitwise operators to build a query.

It's the tooling aspect that's my gripe with it in a string really, it makes it less likely (and more editor-specific) that I can have syntax highlighting, LSP, etc. That's incidentally the only way in which I don't prefer SQL to Django ORM - i.e. it really isn't that it's a DSL (SQL) that bothers me, it's the string.

But hopefully obviously it's not a big deal, I was just surprised at the look of it when I saw it described as 'really nice' and that the project describes itself as 'focussed on ergonomics'. It just doesn't seem brilliant to me, fine perhaps, par for the course apparently, but not remarkable.

Post reply on HN