Live data from Hacker News

Building a new database management system in academia (2017)

cs.cmu.edu

11–20 of 43 posts

Re: Building a new database management system in academia (2017)

#11

This is an announcement from 2017 about "the next five years", which time period is now squarely in the past. Did the DBMS ever come into existence? (If so: link, please). If not: Why should we be interested in this announcement in 2023?

DeepDive from he listed become SnorkelAI, which become hot lately

Re: Building a new database management system in academia (2017)

#13
post #4
post #3

Earlier quoted context omitted.

This is Andy Pavlo, so he probably got sidetracked with https://ottertune.com/ Not sure what op's intention with this was

Actually, it was a combination of three things: 1. OtterTune Start-up ( https://ottertune.com ) 2. Biological Daughter ( https://twitter.com/andy_pavlo/status/1187841279260004355 ) 3. Pandemic When the pandemic first started, I had a bunch of CMU students reach out to me saying that their summer internships were rescinded and that they were looking for a project to work on so that they wouldn't have a gap in their CV…

> Our future project is going to start with the optimizer first.

What's your opinion of recent attempts like LingoDB, that move the query optimizer into a traditional compiler stack, in this case, MLIR?

Re: Building a new database management system in academia (2017)

#15
post #4

Earlier quoted context omitted.

Actually, it was a combination of three things: 1. OtterTune Start-up ( https://ottertune.com ) 2. Biological Daughter ( https://twitter.com/andy_pavlo/status/1187841279260004355 ) 3. Pandemic When the pandemic first started, I had a bunch of CMU students reach out to me saying that their summer internships were rescinded and that they were looking for a project to work on so that they wouldn't have a gap in their CV…

> Our future project is going to start with the optimizer first. What's your opinion of recent attempts like LingoDB, that move the query optimizer into a traditional compiler stack, in this case, MLIR?

LingoDB is an interesting system. Jana has done great work with it. I like projects that take unorthodox approaches to old problems.

The problem with (most) query optimizers is that they take a one shot approach at optimization. I think an optimizer should be built from the groundup to support adaptive query optimization. Something similar to Berkeley's Eddies project from 20 years ago.

Re: Building a new database management system in academia (2017)

#16

If you're interested in the idea of databases built from scratch since the time this post was written in 2017 (based on GitHub contributions info), here are a few: - Materialize: 2017 - DuckDB: 2018 - RedPanda: 2019 - TigerBeetle: 2020

I am building a new immutable cryptographically verified database using IPLD data structures and prolly trees. This allows changes made anywhere to be transparently synced, and for operations to be commuted amongst untrusted peers, for instance allowing for shared index maintenance.

https://use-fireproof.com/docs/architecture

It's also the easiest way to write React apps. Here are some ChatGPT expert builders that I've trained to use the CSS framework of your choice with Fireproof: https://use-fireproof.com/docs/chatgpt-quick-start/#react-ex...

Re: Building a new database management system in academia (2017)

#17
post #4
post #3

Earlier quoted context omitted.

This is Andy Pavlo, so he probably got sidetracked with https://ottertune.com/ Not sure what op's intention with this was

Actually, it was a combination of three things: 1. OtterTune Start-up ( https://ottertune.com ) 2. Biological Daughter ( https://twitter.com/andy_pavlo/status/1187841279260004355 ) 3. Pandemic When the pandemic first started, I had a bunch of CMU students reach out to me saying that their summer internships were rescinded and that they were looking for a project to work on so that they wouldn't have a gap in their CV…

I'm curious, wondering if you could explicate why you feel starting from the query optimization end is key? I have my (amateur) guesses, but would love to hear your expert opinion.

Re: Building a new database management system in academia (2017)

#18
For unsuspecting readers: this article talks about the feasibility of building a new practical DBMS. Database is the most critical piece of software for businesses, so it has already been thoroughly explored and researched. It's very difficult to find a better solution for existing problems. One should either invent a new paradigm or tackle unsolved problems to justify the cost of development.

Technology-wise, writing a toy DBMS is nothing difficult. Even undergraduates can do it.

Re: Building a new database management system in academia (2017)

#19

If you're interested in the idea of databases built from scratch since the time this post was written in 2017 (based on GitHub contributions info), here are a few: - Materialize: 2017 - DuckDB: 2018 - RedPanda: 2019 - TigerBeetle: 2020

According to my estimation, a new database engine is born every week - mostly key-value and document databases. Only a small subset of them survive after one year. According to a guess by Stonebreaker, a DBMS takes around 7 years to become mature enough for general applications.

Re: Building a new database management system in academia (2017)

#20

If you're interested in the idea of databases built from scratch since the time this post was written in 2017 (based on GitHub contributions info), here are a few: - Materialize: 2017 - DuckDB: 2018 - RedPanda: 2019 - TigerBeetle: 2020

According to my estimation, a new database engine is born every week - mostly key-value and document databases. Only a small subset of them survive after one year. According to a guess by Stonebreaker, a DBMS takes around 7 years to become mature enough for general applications.

> According to my estimation, a new database engine is born every week

Fair. I'm talking about databases with funding backing them (either by universities or otherwise).

Post reply on HN