Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a new database management system in academia (2017)

cs.cmu.edu

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Building a new database management system in academia (2017)

#31

Post by lmwnshn »

tlarkworthy wrote: post #25
eatonphil wrote: post #14

If you're interested in the idea of databases built from scratch since the time this post was written in 2017 (based on GitHub contributions info), here are a few: - Materialize: 2017 - DuckDB: 2018 - RedPanda: 2019 - TigerBeetle: 2020

and DuckDB came out of academia too and is not based on Postgres either (highly relevant and notably absent in the authors list of academic DBs at the end of the article) https://duckdb.org/pdf/SIGMOD2019-demo-duckdb.pdf EDIT: oh the article is old

You may be interested in DuckDB's CIDR talk on the little miracles that made it possible. [0]

[0] https://twitter.com/motherduck/status/1615487300523429889

Re: Building a new database management system in academia (2017)

#32

Post by apavlo »

lifepillar wrote: post #27
apavlo wrote: post #4

Earlier quoted context omitted.

Actually, it was a combination of three things: 1. OtterTune Start-up ( https://ottertune.com ) 2. Biological Daughter ( https://twitter.com/andy_pavlo/status/1187841279260004355 ) 3. Pandemic When the pandemic first started, I had a bunch of CMU students reach out to me saying that their summer internships were rescinded and that they were looking for a project to work on so that they wouldn't have a gap in their CV…

I see that MVCC is still your preferred way of doing CC, and what academic research is mostly focused. I am wondering whether that’s an advantage for in-memory databases specifically. I was once discussing MVCC vs 2PL with an experienced Sybase and SQL Server guy, and he claimed that, when transactions are implemented properly and the database is well-designed (no surrogate keys, in particular), 2PL leads to better p…

See our 2014 paper on evaluating CC protocols on in-memory system with high contention / core counts:

https://www.vldb.org/pvldb/vol8/p209-yu.pdf

All the protocols regress to the same. This evaluation was only with stored procedures though. It would be worth doing a similar investigation with conversational DB protocols (e.g., JDBC, ODBC).

Re: Building a new database management system in academia (2017)

#33

Post by zachmu »

eatonphil wrote: post #14

If you're interested in the idea of databases built from scratch since the time this post was written in 2017 (based on GitHub contributions info), here are a few: - Materialize: 2017 - DuckDB: 2018 - RedPanda: 2019 - TigerBeetle: 2020

Dolt started in 2018: https://doltdb.com

Yes we have commit history from 2015 but that's from an earlier db project (noms) that we forked and built on top of

Re: Building a new database management system in academia (2017)

#34

Post by otoolep »

eatonphil wrote: post #14

If you're interested in the idea of databases built from scratch since the time this post was written in 2017 (based on GitHub contributions info), here are a few: - Materialize: 2017 - DuckDB: 2018 - RedPanda: 2019 - TigerBeetle: 2020

I started rqlite[1] in 2014[2], FWIW. While I didn't build the storage engine, or the consensus system, I've built the entire "management" part of the RDBMS from scratch. I'm almost 10 years at it, and there is still plenty to do.

[1] https://www.rqlite.io

[2] https://www.philipotoole.com/9-years-of-open-source-database...

Re: Building a new database management system in academia (2017)

#35

Post by yingjunwu »

Proud to see my name (https://twitter.com/YingjunWu) mentioned in Andy's blog. I was Andy's visiting PhD at CMU and was the top 1 contributor to Peloton (https://github.com/cmu-db/peloton).

Today, building a database from scratch is extremely difficult, for several reasons: 1. it anyways takes a long time; 2. there are so many successful (open-source) databases; 3. hiring top engineers are so expensive. 4. you won't get enough attention unless your system is drastically better than existing ones.

An interesting observation is that very few database was built since 2020 - almost all the newly built databases were developed on top of existing databases (PostgreSQL, ClickHouse, etc).

I started building RisingWave (https://github.com/risingwavelabs/risingwave) in early 2021. The only reason we built the system from scratch was that none of the existing systems can address the problem we are solving - distributed SQL stream processing at cloud scale. We tried Flink but gave up, as it's too heavy and it's architecture was not designed for the cloud environment.

If you want to build a database from scratch, or are simply interested in databases, we may talk.

Re: Building a new database management system in academia (2017)

#36

Post by iudqnolq »

yingjunwu wrote: post #35

Proud to see my name ( https://twitter.com/YingjunWu ) mentioned in Andy's blog. I was Andy's visiting PhD at CMU and was the top 1 contributor to Peloton ( https://github.com/cmu-db/peloton ). Today, building a database from scratch is extremely difficult, for several reasons: 1. it anyways takes a long time; 2. there are so many successful (open-source) databases; 3. hiring top engineers are so expensive. 4. you wo…

You're obviously the expert here, but I was surprised that you found it notable very few databases have been released in the last three years. That seems like a very short timeframe. Per Wikipedia ClickHouse started as an experimental project in 2009 and was first released in 2016.

Re: Building a new database management system in academia (2017)

#37

Post by sundar28 »

apavlo wrote: post #32
lifepillar wrote: post #27

Earlier quoted context omitted.

I see that MVCC is still your preferred way of doing CC, and what academic research is mostly focused. I am wondering whether that’s an advantage for in-memory databases specifically. I was once discussing MVCC vs 2PL with an experienced Sybase and SQL Server guy, and he claimed that, when transactions are implemented properly and the database is well-designed (no surrogate keys, in particular), 2PL leads to better p…

See our 2014 paper on evaluating CC protocols on in-memory system with high contention / core counts: https://www.vldb.org/pvldb/vol8/p209-yu.pdf All the protocols regress to the same. This evaluation was only with stored procedures though. It would be worth doing a similar investigation with conversational DB protocols (e.g., JDBC, ODBC).

Thanks for the reminder.. this was a great paper, and I'm curious - why would conversational protocols indicate any difference? Also - curious if any such tests have used libraries such as seastar?

Re: Building a new database management system in academia (2017)

#38

Post by frankdejonge »

eatonphil wrote: post #14

If you're interested in the idea of databases built from scratch since the time this post was written in 2017 (based on GitHub contributions info), here are a few: - Materialize: 2017 - DuckDB: 2018 - RedPanda: 2019 - TigerBeetle: 2020

Correct me if I’m wrong, but I don’t think RedPanda is a database. I see it as a streaming data solution, which the novelty factor can be discussed as well, since it’s basically Kafka.

Re: Building a new database management system in academia (2017)

#39

Post by erichocean »

zinclozenge wrote: post #30
erichocean wrote: post #13

Earlier quoted context omitted.

> Our future project is going to start with the optimizer first. What's your opinion of recent attempts like LingoDB, that move the query optimizer into a traditional compiler stack, in this case, MLIR?

There's also mutable that compiles to WASM and lets it get JITed by v8 https://github.com/mutable-org/mutable .

Wow, good find! Thanks for the link.

Re: Building a new database management system in academia (2017)

#40

Post by erichocean »

apavlo wrote: post #15
erichocean wrote: post #13

Earlier quoted context omitted.

> Our future project is going to start with the optimizer first. What's your opinion of recent attempts like LingoDB, that move the query optimizer into a traditional compiler stack, in this case, MLIR?

LingoDB is an interesting system. Jana has done great work with it. I like projects that take unorthodox approaches to old problems. The problem with (most) query optimizers is that they take a one shot approach at optimization. I think an optimizer should be built from the groundup to support adaptive query optimization. Something similar to Berkeley's Eddies project from 20 years ago.

One way to do that might be to merge the query optimizer and executor together and then execute them simultaneously in a self-adjusting code framework.[0] The relevant optimizer variables would then be set initially (using whatever mechanism already exists e.g. heuristics, stats, etc.) and then while the query executes, you continually update those optimizer variables. The self-adjusting code property would cause the query to self-adjust as it ran, while still producing the same end result.

I'm sure there are details I'm missing here, but I do believe the general approach could do implemented in LingoDB (or similar) as a compiler transformation, so the actually cost-to-develop this approach would remain tractable.

[0] I suspect you'd need to model the whole thing as a streaming network so that you can update the network parts as you go, effectively re-wiring the streams while not invalidating earlier results. So SAC+logic to map from one stream architecture to another. JITs that support de-optimization have to do something similar (with a lot of careful upfront design), so that's at least plausible.

Post reply on HN