anyone knows something similar for a graph database ?
SimpleDB: A Basic RDBMS Built from Scratch
41–50 of 53 posts
Re: SimpleDB: A Basic RDBMS Built from Scratch
#42A quick question about the MIT course this student followed[1]. Am I correct in assuming that the way to follow this course would be to take the lecture notes, the assigned reading material, and simply read it before then moving on to the assignments? Just wanted to make sure I'm not missing some video/other content before I take a look into this stuff. [1] https://ocw.mit.edu/courses/electrical-engineering-and-compu…
I think that is the recommended approach, but I just read the lecture notes and started working on the assignments. I tried to prioritize coding because it helps me retain knowledge better over time
Appreciate you sharing your direct experience. Thank you!
Re: SimpleDB: A Basic RDBMS Built from Scratch
#43A quick question about the MIT course this student followed[1]. Am I correct in assuming that the way to follow this course would be to take the lecture notes, the assigned reading material, and simply read it before then moving on to the assignments? Just wanted to make sure I'm not missing some video/other content before I take a look into this stuff. [1] https://ocw.mit.edu/courses/electrical-engineering-and-compu…
How does this stack up against Intro to Database Systems from CMU?
When I was implementing SimpleDB in 2019, I believe CMU's course didn't have resources and lab assignments that were publicly available. Now CMU has published a full video lecture series (which MIT doesn't have) and their labs. So if I were starting again today, I would probably go with CMU's course.
CMU Intro to Databases Labs: https://15445.courses.cs.cmu.edu/fall2021/assignments.html
CMU Intro to Databases Lectures: https://www.youtube.com/playlist?list=PLSE8ODhjZXjZaHA6QcxDf...
BusTub - CMU's Version of SimpleDB: https://github.com/cmu-db/bustub
Re: SimpleDB: A Basic RDBMS Built from Scratch
#44It's cool to see this here, thanks for posting! I remember working with SimpleDB when taking a a database internals class. It was an interesting and engaging way to learn concepts like strict two-phase locking and Selinger-style query optimization.
Yeah, I think its a shame that most teachers don't give assignments like this that tie the big picture together with the low-level details. After students complete a big assignment like SimpleDB, they'll have a working artifact that they can reference for the rest of their career
Re: SimpleDB: A Basic RDBMS Built from Scratch
#45I love seeing these educational databases. I'm still waiting for one to implement some form of MVCC, every one so far always seems to use 2 phase locking.
What other educational databases are you looking at? Curious in case I'm missing any of them.
There is also BusTub from CMU which I stumbled upon earlier today:
Re: SimpleDB: A Basic RDBMS Built from Scratch
#46OP I am curious about the project license. Since it is built on top of the assignment code, can you use your own copyright? - https://github.com/awelm/simpledb/blob/2e78bb2/LICENSE
Good point. I just removed my name from the license, but I couldn't find the original author(s) so the copyright name list is just empty now
Re: SimpleDB: A Basic RDBMS Built from Scratch
#47Reminded me of ToyDB, written in Rust. https://github.com/erikgrinaker/toydb
Re: SimpleDB: A Basic RDBMS Built from Scratch
#48Great work. I'm not an expert in database internals, but maybe someone can help answer some of these:
- Why not just use mmap and let the OS handle page caching?
- Why not use a write-ahead-log for all writes, with a background thread applying transactions to the read replica asynchronously? In most cases eventual consistency is fine and you don't need to query the latest version of the data, but this can be an optional query parameter.
- Instead of embedding an unpredictable SQL-to-code compiler, why not provide direct access to physical (relational algebra) operators via function calls? eg let me write: select([fields]).where(x=2).join(table) etc ... letting me select which index to use, how to do the joins etc.
Re: SimpleDB: A Basic RDBMS Built from Scratch
#49I love seeing these educational databases. I'm still waiting for one to implement some form of MVCC, every one so far always seems to use 2 phase locking.
LMDB implements MVCC and is pretty easy to read, the implementation is very tiny.
Re: SimpleDB: A Basic RDBMS Built from Scratch
#50Great work. I'm not an expert in database internals, but maybe someone can help answer some of these: - Why not just use mmap and let the OS handle page caching? - Why not use a write-ahead-log for all writes, with a background thread applying transactions to the read replica asynchronously? In most cases eventual consistency is fine and you don't need to query the latest version of the data, but this can be an optio…
Great questions! I'm not a database expert either but I can try answering these:
1) I think databases like to manage pages directly because the db can make more optimizations than the OS because the db has more context. For example, when aborting a transaction the db knows its dirty pages should be evicted (i'm not sure if mmap offers custom eviction). Also I believe if the db uses mmap, it loses control over when pages are flushed to disk. Flush control is necessary for guaranteeing transaction durability.
2) What you're describing here sounds similar to a LSM-tree database (e.g. RocksDB). They are used often for write-heavy workloads because writes are just appends, but they might not be great for read-heavy things.
3) This reminds me of PRQL[1] (which was trending on Hacker News last week) and Spark SQL. I'm not too familiar with this area though, so I can't really say why SQL was designed this way.
[1] https://github.com/max-sixty/prql?utm_source=hackernewslette...