Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a SQL database from scratch in Go

notes.eatonphil.com

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Writing a SQL database from scratch in Go

#31

Post by arendtio »

cultofmetatron wrote: post #24
alperakgun wrote: post #23

Curious - would Rust be more appropriate than Go for such a task?

rust would be more appropriate if your intention was to use this in production. Databases ad GC don't mix well. (its done but it makes tuning a nightmare) As much as I love rust, its learning curve is high and I'm sure Op doesn't want to spend half his article teaching all the intricacies of types and the borrow checker. Go is easy to learn over a weekend so its probably a better medium for illustrating the concepts…

Sometimes I wonder how Go would look like with a borrow checker...

Re: Writing a SQL database from scratch in Go

#32

Post by tbrock »

cube2222 wrote: post #20

Great blog post! I'd just like to add for the curious, that usually you'd use goyacc for parsing SQL. And most serious SQL projects in Go have started with the SQL parser from vitess and adapted it to their use case (which is just funny trivia, but for anything big, I recommend it, did the same for OctoSQL [0]). [0]: https://github.com/cube2222/octosql

Why did all of the Golang SQL parsers come from Vitess? I would love to know more about this history.

Was it because they were the first and people just started using it or is it the best for some reason?

Re: Writing a SQL database from scratch in Go

#33

Post by gpetukhov »

throwlaplace wrote: post #4
zoom6628 wrote: post #3

Probably the first article Ive ever read about lexical parsing with code that i have actually understood - and I dont even program in golang. Great job.

Lexical analysis using these bespoke methods (writing the finite state machine) is so tedious and error prone. I don't have that much experience but I just went through crafting interpreters and replaced this same module with https://github.com/J-F-Liu/pom , which is a parser combinator library, and it was way easier.

I guess it's relevant.

Here is a great set of videos from "Low Level Javascript" channel explaining how to write parser combinators from scratch. - https://www.youtube.com/watch?v=6oQLRhw5Ah0&list=PLP29wDx6Qm...

Re: Writing a SQL database from scratch in Go

#34

Post by eatonphil »

bogomipz wrote: post #17
eatonphil wrote: post #2

Unexpected, but welcome, to see the first part in the series on here right now. I just published the second part [0] today, featuring binary expressions and WHERE filtering. It also updates the REPL to use a prettier table-printing library and a readline implementation. The repo [1] has some additional bare notes on architecture and links to similar, more mature projects (primarily go-mysql-server and ramsql). [0] ht…

This is great. Do you plan on continuing this blog series? If so I hope you post it here. Cheers.

Definitely! I typically do post here. There are also links for various ways to subscribe on the site itself (e.g. twitter, email).

Re: Writing a SQL database from scratch in Go

#35

Post by eatonphil »

cube2222 wrote: post #20

Great blog post! I'd just like to add for the curious, that usually you'd use goyacc for parsing SQL. And most serious SQL projects in Go have started with the SQL parser from vitess and adapted it to their use case (which is just funny trivia, but for anything big, I recommend it, did the same for OctoSQL [0]). [0]: https://github.com/cube2222/octosql

I like this project! I think I've asked you about it supporting parquet before... I'll add it to my list of similar projects.

Regarding goyacc and "usually", I myself was curious about this years ago and asked on Stack Exchange: Do modern languages still use parser generators? [0].

Most of the time, major languages _do_ use hand-written parsers.

Of course it may literally be the case that most other SQL projects in Go do use goyacc. My point is that in general, hand-written parsers are more common in real software than you might think.

[0] https://softwareengineering.stackexchange.com/questions/2502...

Re: Writing a SQL database from scratch in Go

#36

Post by eatonphil »

tbrock wrote: post #32
cube2222 wrote: post #20

Great blog post! I'd just like to add for the curious, that usually you'd use goyacc for parsing SQL. And most serious SQL projects in Go have started with the SQL parser from vitess and adapted it to their use case (which is just funny trivia, but for anything big, I recommend it, did the same for OctoSQL [0]). [0]: https://github.com/cube2222/octosql

Why did all of the Golang SQL parsers come from Vitess? I would love to know more about this history. Was it because they were the first and people just started using it or is it the best for some reason?

SQL is a humongous spec. Any serious project would rather piggy-back off an existing parser. Most of the interesting parts for most people is implementing backends against in-memory, disk, S3, HDFS, etc.

Re: Writing a SQL database from scratch in Go

#37

Post by throwaway894345 »

userbinator wrote: post #13
ycnewsreader wrote: post #8

Earlier quoted context omitted.

Another reason is that by providing an identifier for this string literal, misspellings of it can be detected by the compiler whereas there is nothing tying separate string literals together.

I can hardly imagine a case where you would misspell "select" and not notice it at some point, nor use it in more than one place (the keyword detector) in the parser.

It’s one of the more common bugs in our office. Before we integrated a Python linter in our CI for a prototype project, we would see several such typo errors each week with a small team (and that was with symbols, not string literals).

Re: Writing a SQL database from scratch in Go

#38

Post by cube2222 »

tbrock wrote: post #32
cube2222 wrote: post #20

Great blog post! I'd just like to add for the curious, that usually you'd use goyacc for parsing SQL. And most serious SQL projects in Go have started with the SQL parser from vitess and adapted it to their use case (which is just funny trivia, but for anything big, I recommend it, did the same for OctoSQL [0]). [0]: https://github.com/cube2222/octosql

Why did all of the Golang SQL parsers come from Vitess? I would love to know more about this history. Was it because they were the first and people just started using it or is it the best for some reason?

No idea about the history.

From my perspective (as the author of OctoSQL): I need a SQL parser. Vitess has a fully fledged MySQL compatible one. I'd probably want to copy the open source vitess one and add the necessary features myself, instead of writing it from scratch. It's just a big endeavor, a few thousand lines of goyacc code and even more Go boilerplate.

I'm not saying everybody is using the same library, because the code has been heavily modified after being copied. A look at the cockroachdb parses suffices to see how much changed it is.

If you look at OctoSQL, there too have been many changes to add stuff like table valued functions.

Re: Writing a SQL database from scratch in Go

#39

Post by eatonphil »

cultofmetatron wrote: post #24
alperakgun wrote: post #23

Curious - would Rust be more appropriate than Go for such a task?

rust would be more appropriate if your intention was to use this in production. Databases ad GC don't mix well. (its done but it makes tuning a nightmare) As much as I love rust, its learning curve is high and I'm sure Op doesn't want to spend half his article teaching all the intricacies of types and the borrow checker. Go is easy to learn over a weekend so its probably a better medium for illustrating the concepts…

Truth is I've written about how to parse in better languages before (JavaScript, Python, Standard ML, etc.) and I wanted to figure out a good approach for doing it in Go.

As for databases and GC, you may be right but there are still major databases out there written in Java for example.

https://www.quora.com/Which-databases-data-stores-are-writte...

Post reply on HN