Show HN: An experimental distributed SQL database from scratch in Go
1–10 of 15 posts
Re: Show HN: An experimental distributed SQL database from scratch in Go
#2Hey HN,
Work is coming along on this project to build a distributed SQL database from scratch, mostly as a reference for newcomers to get an idea about the inner workings.
Looking for contributors who are interested in anything from parsers, disk paging, building out a REPL, defining an IR grammar, implementing consensus and more!
Anyone interested in contributing in these areas is more than welcome!
Re: Show HN: An experimental distributed SQL database from scratch in Go
#3Hey HN, Work is coming along on this project to build a distributed SQL database from scratch, mostly as a reference for newcomers to get an idea about the inner workings. Looking for contributors who are interested in anything from parsers, disk paging, building out a REPL, defining an IR grammar, implementing consensus and more! Anyone interested in contributing in these areas is more than welcome!
It takes 5-10 years to write, test and productionize a database or filesystem. If you're planning to invest that kind of time and effort, some suggestions are:
* if you're writing a distributed database, a novel and valuable feature would be to consider the network partition case foremost. For example, design the database from the standpont of a node being down for a month.
* do adequate logging so that an operator can understand what is going right or wrong
* how can terminated nodes automatically be rebuilt efficiently and automatically?
* all configuration settings should be dynamic
Source: experienced DBA, worked with Cassandra, Influxdb and most SQL RDBMSs
Re: Show HN: An experimental distributed SQL database from scratch in Go
#4Hey HN, Work is coming along on this project to build a distributed SQL database from scratch, mostly as a reference for newcomers to get an idea about the inner workings. Looking for contributors who are interested in anything from parsers, disk paging, building out a REPL, defining an IR grammar, implementing consensus and more! Anyone interested in contributing in these areas is more than welcome!
How are the goals of this project different than cockroachdb?
Re: Show HN: An experimental distributed SQL database from scratch in Go
#5Hey HN, Work is coming along on this project to build a distributed SQL database from scratch, mostly as a reference for newcomers to get an idea about the inner workings. Looking for contributors who are interested in anything from parsers, disk paging, building out a REPL, defining an IR grammar, implementing consensus and more! Anyone interested in contributing in these areas is more than welcome!
It takes 5-10 years to write, test and productionize a database or filesystem. If you're planning to invest that kind of time and effort, some suggestions are: * if you're writing a distributed database, a novel and valuable feature would be to consider the network partition case foremost. For example, design the database from the standpont of a node being down for a month. * do adequate logging so that an operator c…
> if you're writing a distributed database
is this really a requirement for all distributed databases or only decentralized databases? if I'm using a distributed database and I control all the nodes, if one of them is down I don't plan on spinning it back up (assuming the data is replicated across other nodes). I guess what I'm saying is if I need 20 nodes, I'm going to make sure I always have 20 nodes running.
Re: Show HN: An experimental distributed SQL database from scratch in Go
#6Hey HN, Work is coming along on this project to build a distributed SQL database from scratch, mostly as a reference for newcomers to get an idea about the inner workings. Looking for contributors who are interested in anything from parsers, disk paging, building out a REPL, defining an IR grammar, implementing consensus and more! Anyone interested in contributing in these areas is more than welcome!
It takes 5-10 years to write, test and productionize a database or filesystem. If you're planning to invest that kind of time and effort, some suggestions are: * if you're writing a distributed database, a novel and valuable feature would be to consider the network partition case foremost. For example, design the database from the standpont of a node being down for a month. * do adequate logging so that an operator c…
> design the database from the standpont of a node being down for a month.
It uses Raft, so this should be handled by nature. If you are referring to writes to said node (not that Raft would allow it), you are delving into CAP theorem and what you are suggesting is impossible (unless you don't care about consistency).
Re: Show HN: An experimental distributed SQL database from scratch in Go
#7I am embarking upon a similar project: I’m building a privacy-focused product similar to Google Photos and Flickr. My goal is to not only to develop a usable product, but also to document the process in as much detail as possible in order to enable others to build similar things.
I feel strongly about spreading the skills required to build non-trivial products. I hope to enable others to deliver similar technical projects.
Re: Show HN: An experimental distributed SQL database from scratch in Go
#8I am embarking upon a similar project: I’m building a privacy-focused product similar to Google Photos and Flickr. My goal is to not only to develop a usable product, but also to document the process in as much detail as possible in order to enable others to build similar things. I feel strongly about spreading the skills required to build non-trivial products. I hope to enable others to deliver similar technical pro…
That’s awesome, I look forward to your forthcoming Show HN.
Re: Show HN: An experimental distributed SQL database from scratch in Go
#9Hey HN, Work is coming along on this project to build a distributed SQL database from scratch, mostly as a reference for newcomers to get an idea about the inner workings. Looking for contributors who are interested in anything from parsers, disk paging, building out a REPL, defining an IR grammar, implementing consensus and more! Anyone interested in contributing in these areas is more than welcome!
If it is intended to be a reference project, I suggest focusing on writing the README sections first highlighting the main areas and then linking to code sections to ease the navigation.
You may or may not be aware, but Andy Pavlo records his courses and puts them on YouTube. His latest playlist covers the main database topics with the last five or so lectures covering distributed databases: https://www.youtube.com/playlist?list=PLSE8ODhjZXjbohkNBWQs_...
edit: ^suggesting Pavlo's work since he introduces database concepts very well, so it may be worth structuring the reference architecture in the same way.
Re: Show HN: An experimental distributed SQL database from scratch in Go
#10I am embarking upon a similar project: I’m building a privacy-focused product similar to Google Photos and Flickr. My goal is to not only to develop a usable product, but also to document the process in as much detail as possible in order to enable others to build similar things. I feel strongly about spreading the skills required to build non-trivial products. I hope to enable others to deliver similar technical pro…
Have you seen Perkeep? https://perkeep.org/
Mainly build by Brad Fitzpatrick, in Go.