Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Writing a SQL database from scratch in Go

notes.eatonphil.com

Post reply on HN
Print view View original

21–30 of 55 posts

Re: Writing a SQL database from scratch in Go

#21

Post by papaf »

throwlaplace wrote: post #4
zoom6628 wrote: post #3

Probably the first article Ive ever read about lexical parsing with code that i have actually understood - and I dont even program in golang. Great job.

Lexical analysis using these bespoke methods (writing the finite state machine) is so tedious and error prone. I don't have that much experience but I just went through crafting interpreters and replaced this same module with https://github.com/J-F-Liu/pom , which is a parser combinator library, and it was way easier.

which is a parser combinator library, and it was way easier

I like parser combinators but a word of warning.

A lot of parser combinators (not all) have problems with recursive grammar such as Json and SQL [1].

For instance, a JSON map can contain a JSON map and this can lead to stack overflows when defining the grammar.

[1] https://fsharpforfunandprofit.com/posts/understanding-parser...

Re: Writing a SQL database from scratch in Go

#22

Post by pizza234 »

cube2222 wrote: post #20

Great blog post! I'd just like to add for the curious, that usually you'd use goyacc for parsing SQL. And most serious SQL projects in Go have started with the SQL parser from vitess and adapted it to their use case (which is just funny trivia, but for anything big, I recommend it, did the same for OctoSQL [0]). [0]: https://github.com/cube2222/octosql

This is not the first educational project I've seen that insists on manually writing the parser.

Call me contrarian but I find the article's approach poor (at least, as it is now).

If one looks at the overall content of the two articles of the series, 70%/80% of the content (or possibly more) is parsing, which is arguably the least interesting, or at least, the most boilerplate, part of a database system.

Re: Writing a SQL database from scratch in Go

#24

Post by cultofmetatron »

alperakgun wrote: post #23

Curious - would Rust be more appropriate than Go for such a task?

rust would be more appropriate if your intention was to use this in production. Databases ad GC don't mix well. (its done but it makes tuning a nightmare)

As much as I love rust, its learning curve is high and I'm sure Op doesn't want to spend half his article teaching all the intricacies of types and the borrow checker. Go is easy to learn over a weekend so its probably a better medium for illustrating the concepts as everything is laid out simply.

Re: Writing a SQL database from scratch in Go

#26

Post by _ph_ »

userbinator wrote: post #5

selectKeyword keyword = "select" I don't work with Go, so this may be a requirement of the language that I don't know, but whenever I see lines like this, it automatically brings up the question why? --- do you really expect to need to rename the SELECT keyword? Especially when it's named "selectKeyword". Why not just use the string constant? Ditto for the others like "leftparenSymbol" --- I see there's explicit char…

The biggest reason is typo and type checking. Any typo is a compilation error. But the type checking is even more interesting. You basically create a subtype of string which is correspondingly type checked by the compiler. Functions might return a keyword instead of a string.

One funny thing you can do in Go is for example to create a subtype of int and change its String() method to return the hexadecimal representation of that int. Very neat.

Re: Writing a SQL database from scratch in Go

#27

Post by orhanhh »

speedgoose wrote: post #25
alperakgun wrote: post #23

Curious - would Rust be more appropriate than Go for such a task?

Golang works too. There is a few databases written in Golang, such as Prometheus, InfluxDB, CockroachDB, or tidb. https://github.com/topics/database?l=go&o=desc&s=stars

Note that neither CockroachDB or TiDB use Golang for their actual storage engine, which is in both cases written in C (RocksDB). They do use Golang for SQL parsing though, which is what this post was mostly about.

Re: Writing a SQL database from scratch in Go

#28

Post by nednar »

eatonphil wrote: post #2

Unexpected, but welcome, to see the first part in the series on here right now. I just published the second part [0] today, featuring binary expressions and WHERE filtering. It also updates the REPL to use a prettier table-printing library and a readline implementation. The repo [1] has some additional bare notes on architecture and links to similar, more mature projects (primarily go-mysql-server and ramsql). [0] ht…

The idea is nice, but it seems like the text is not explaining much. If you don't have a background in parser writing you have no clue what types are defined there and why in the very first code block. If you're not an SQL expert you might still know what SELECT and WHERE is, but do you think everybody knows what's an INTO keyword is used for? Where does the lex function come from? Why do I have a lexer and a lex()? Where does the cursor come from that I give into it? Why don't I give it a string? My user input SQL statement is a string, right? Why does the lex function have two for loops? What can't be done in one? Does this whole logic actually have a name maybe? What other alternatives to parsing SQL are out there? Why choose this path?

All that knowledge can't be gathered by just staring at the source code. And for people who understand the source code well enough to gather all this info themselves maybe they don't need the blog post around it, right?

Now all this I don't say to demotivate you. I hope you're strong enough to get over negative feedback and think about rewriting it to something more useful. I see a lot of potential here. But every good writer needs one (or ten million hacker news reading) editors to get to the really good content.

Re: Writing a SQL database from scratch in Go

#29

Post by cube2222 »

orhanhh wrote: post #27
speedgoose wrote: post #25

Earlier quoted context omitted.

Golang works too. There is a few databases written in Golang, such as Prometheus, InfluxDB, CockroachDB, or tidb. https://github.com/topics/database?l=go&o=desc&s=stars

Note that neither CockroachDB or TiDB use Golang for their actual storage engine, which is in both cases written in C (RocksDB). They do use Golang for SQL parsing though, which is what this post was mostly about.

However, cockroachdb does all other work (including query execution) in Go too.

There's also DGraph based on Badger as a storage engine which is an all-go stack.

And badger does compare favorably to rocksdb under certain workloads.

Re: Writing a SQL database from scratch in Go

#30

Post by irfansharif »

orhanhh wrote: post #27
speedgoose wrote: post #25

Earlier quoted context omitted.

Golang works too. There is a few databases written in Golang, such as Prometheus, InfluxDB, CockroachDB, or tidb. https://github.com/topics/database?l=go&o=desc&s=stars

Note that neither CockroachDB or TiDB use Golang for their actual storage engine, which is in both cases written in C (RocksDB). They do use Golang for SQL parsing though, which is what this post was mostly about.

> Note that neither CockroachDB [...] use Golang for their actual storage engine

We do, now. We're looking to move away from RocksDB to https://github.com/cockroachdb/pebble/.

Post reply on HN