SimpleDB: A Basic RDBMS Built from Scratch • Hacker News Board

Re: SimpleDB: A Basic RDBMS Built from Scratch

#51

Post by haggy102 » Wed, Feb 02, 2022, 5:12 PM UTC

hackthesystem wrote: post #44

albertzhong wrote: post #39

It's cool to see this here, thanks for posting! I remember working with SimpleDB when taking a a database internals class. It was an interesting and engaging way to learn concepts like strict two-phase locking and Selinger-style query optimization.

Yeah, I think its a shame that most teachers don't give assignments like this that tie the big picture together with the low-level details. After students complete a big assignment like SimpleDB, they'll have a working artifact that they can reference for the rest of their career

I think the main issue that universities face is time. There is only so much time in each semester and, as we all know, building and improving on a database is a lifelong task.

Re: SimpleDB: A Basic RDBMS Built from Scratch

#52

Post by akrymski » Wed, Feb 02, 2022, 9:54 PM UTC

hackthesystem wrote: post #50

akrymski wrote: post #48

Great work. I'm not an expert in database internals, but maybe someone can help answer some of these: - Why not just use mmap and let the OS handle page caching? - Why not use a write-ahead-log for all writes, with a background thread applying transactions to the read replica asynchronously? In most cases eventual consistency is fine and you don't need to query the latest version of the data, but this can be an optio…

Great questions! I'm not a database expert either but I can try answering these: 1) I think databases like to manage pages directly because the db can make more optimizations than the OS because the db has more context. For example, when aborting a transaction the db knows its dirty pages should be evicted (i'm not sure if mmap offers custom eviction). Also I believe if the db uses mmap, it loses control over when pa…

1) Indeed you should only use mmap for reads afaik

2) Was thinking more of an event-sourcing model, whereby you log the SQL statements first, then update a B-Tree in the background.

Read via mmap, write by appending to a log and asynchronously applying the changes to the file.

3) Rather than yet another QL, expose a higher level API that I can target in any language

Re: SimpleDB: A Basic RDBMS Built from Scratch

#53

Post by akrymski » Thu, Feb 03, 2022, 1:05 PM UTC

akrymski wrote: post #52

hackthesystem wrote: post #50

Earlier quoted context omitted.

Great questions! I'm not a database expert either but I can try answering these: 1) I think databases like to manage pages directly because the db can make more optimizations than the OS because the db has more context. For example, when aborting a transaction the db knows its dirty pages should be evicted (i'm not sure if mmap offers custom eviction). Also I believe if the db uses mmap, it loses control over when pa…

1) Indeed you should only use mmap for reads afaik 2) Was thinking more of an event-sourcing model, whereby you log the SQL statements first, then update a B-Tree in the background. Read via mmap, write by appending to a log and asynchronously applying the changes to the file. 3) Rather than yet another QL, expose a higher level API that I can target in any language

Another thing to consider is pluggable storage (a key/value interface) and pluggable query language (relational algebra interface?) and how to fit the two together.