41–50 of 55 posts
Earlier quoted context omitted.
Note that neither CockroachDB or TiDB use Golang for their actual storage engine, which is in both cases written in C (RocksDB). They do use Golang for SQL parsing though, which is what this post was mostly about.
> Note that neither CockroachDB [...] use Golang for their actual storage engine We do, now. We're looking to move away from RocksDB to https://github.com/cockroachdb/pebble/ .
Earlier quoted context omitted.
The pattern (which I employ only sometimes) is to have almost all literals defined in this way. Perhaps SELECT isn’t likely to be the word you misspell, but I’m a careless typist and make 10 typos a minute. Taking this added step helps your editor save you from yourself.
Having standards like that and keeping them helps a lot. Next time you have a different keyword, you don't have to think "does it deserve a constant?" - all of them do. Similar to how linters stop you from overthinking indentation in specific cases, or some naming standards, and later rename/reformat-wars.
That's what leads to dogmatic cargo-culting. Good software is written by thinking about the circumstances and doing what makes the most sense, not by mindless rule-following that don't always make sense.
I don't know why someone would be so worried about typos and introduce more verbosity and redundancy in the process; but then again, I don't use an IDE and I've never had this problem.
Earlier quoted context omitted.
Why did all of the Golang SQL parsers come from Vitess? I would love to know more about this history. Was it because they were the first and people just started using it or is it the best for some reason?
SQL is a humongous spec. Any serious project would rather piggy-back off an existing parser. Most of the interesting parts for most people is implementing backends against in-memory, disk, S3, HDFS, etc.
https://www.reddit.com/r/golang/comments/fgwwlx/database_bas...
Great blog post! I'd just like to add for the curious, that usually you'd use goyacc for parsing SQL. And most serious SQL projects in Go have started with the SQL parser from vitess and adapted it to their use case (which is just funny trivia, but for anything big, I recommend it, did the same for OctoSQL [0]). [0]: https://github.com/cube2222/octosql
I like this project! I think I've asked you about it supporting parquet before... I'll add it to my list of similar projects. Regarding goyacc and "usually", I myself was curious about this years ago and asked on Stack Exchange: Do modern languages still use parser generators? [0]. Most of the time, major languages _do_ use hand-written parsers. Of course it may literally be the case that most other SQL projects in G…
Regarding the parsing part, that's interesting. Though the cockroachdb parser is a _great_ lecture about how to add nice error messages with goyacc. (You add a special error token which captures anything after a syntax error, so the parser doesn't break down. At least something around this, haven't yet dived deep into it.)
Though I know Go itself uses a handwritten parser.
Earlier quoted context omitted.
I can hardly imagine a case where you would misspell "select" and not notice it at some point, nor use it in more than one place (the keyword detector) in the parser.
It’s one of the more common bugs in our office. Before we integrated a Python linter in our CI for a prototype project, we would see several such typo errors each week with a small team (and that was with symbols, not string literals).
Earlier quoted context omitted.
Lexical analysis using these bespoke methods (writing the finite state machine) is so tedious and error prone. I don't have that much experience but I just went through crafting interpreters and replaced this same module with https://github.com/J-F-Liu/pom , which is a parser combinator library, and it was way easier.
which is a parser combinator library, and it was way easier I like parser combinators but a word of warning. A lot of parser combinators (not all) have problems with recursive grammar such as Json and SQL [1]. For instance, a JSON map can contain a JSON map and this can lead to stack overflows when defining the grammar. [1] https://fsharpforfunandprofit.com/posts/understanding-parser...
Earlier quoted context omitted.
(it’s actually also straightforward to write a recursive descent parser directly on the char stream w/o a lexing step.)
how? it's not like you can magically skip the actual tokenization. you're basically saying you can do lexical analysis and semantic analysis in the same function. sure but that makes the code that much hairier - there's a reason why they're typically factored into a lexer and a parser.
yes
> that makes the code that much hairier
true, but it does save you the rigamarole of the lexer.
func parseDeclaration(s string, i0 Pos) (n Node, i Pos, err error) {
i = i0
i = skipSpace(s, i)
if word(s, i, "var") {
v := &ast.VarDecl{VarPos: i}
i += 3
i = skipSpace(s, i)
if v.Ident, i, err = parseIdent(s, i); err != nil {
return
}
i = skipSpace(s, i)
if is(s, i, "=") {
i += 1
i = skipSpace(s, i)
if v.Value, i, err = parseExpr(s, i); err != nil {
return
}
}
if !is(s, i, ";") {
return nil, i, fmt.Errorf(`unexpected input or EOF (expected ";")`)
}
⋮Earlier quoted context omitted.
It’s one of the more common bugs in our office. Before we integrated a Python linter in our CI for a prototype project, we would see several such typo errors each week with a small team (and that was with symbols, not string literals).
Python and a lot of the other dynamic languages are in a slightly different situation.
selectKeyword keyword = "select" I don't work with Go, so this may be a requirement of the language that I don't know, but whenever I see lines like this, it automatically brings up the question why? --- do you really expect to need to rename the SELECT keyword? Especially when it's named "selectKeyword". Why not just use the string constant? Ditto for the others like "leftparenSymbol" --- I see there's explicit char…
For example when storing session state into a Dictionary all access to the session state is through static strings, never a string literal. This makes is super easy to find everywhere that’s accessing that particular session variable.