Live data from Hacker News

SQLite Release 3.25.0 adds support for window functions

sqlite.org

81–90 of 116 posts

Re: SQLite Release 3.25.0 adds support for window functions

#81
Unrelated to window functions, but I finally took the time to start digging into SQLite's internals. People always sing its praises, so it was time to see what all the fuss was about.

Someone else already mentioned that the vast majority of SQLite's codebase are tests. Well, on top of that, of the real "working" codebase I'd say the majority of it is comments. It's incredible. The source is more book than code. If you have any when, why, or how question about SQLite, I guarantee it's answered in the code comments (or at least one of the hundreds of superb documents on their website).

Another surprise I discovered: SQLite has a virtual machine and its own bytecode. All queries you execute against a SQLite database are compiled into SQLite's own little bytecode and then executed on a VM designed for working with SQLite's database. Go ahead, start `sqlite3 yourdb.sqlite` and then run `explain select * from yourtable;`. It'll dump the bytecode for that statement; or any statement you put after `explain`. So cool!

In hindsight, it makes a lot of sense, and a well built VM can be nearly as efficient as any other alternative.

https://sqlite.org/arch.html https://sqlite.org/opcode.html

Fun bit of history. The VM used to be stack based, but now it's register based. I guess they learned the same lessons the rest of the industry learned over that time period :P

(N.B. the bytecode is for internal use only; it's not a public facing API. You should never, ever use bytecode directly yourself.)

There are some painful parts of the codebase though. These aren't "cons" per se. More like necessarily evils.

1) It is filled to the brim with backwards compatibility hacks that make the code more complex than it strictly needs to be. (Most of these are the result of various users of the library misusing the API. The SQLite devs are generous enough to grandfather in the bugs that made those applications work. That's excellent, but it definitely makes the code more "crusty".)

2) One of SQLite's big features is its flexible memory subsystem. It handles OOM, and provides an API for completely customizing the memory subsystem. But given that this is C and memory allocation and interaction is pervasive, the code ends up littered with function calls and clauses. Handling OOM is no small task, and often how to handle the OOM is different in different places. So you can imagine the complexity that adds to the codebase.

Again, those are necessary evils, so its not something I'm "complaining" about. But I thought they were worth mentioning for fellow adventures like me who decide to dive in (which I highly recommend).

So, thanks to how well designed SQLite is overall, and their great documentation, I was able to write a parser in Rust for the SQLite file format in a handful of hours (https://sqlite.org/fileformat2.html). The file format is surprisingly simple. I'm now writing a Cursor to walk the tables, which is a fun exercise of classic B-Tree algorithms.

Re: SQLite Release 3.25.0 adds support for window functions

#83
post #79
post #58

The query optimizer improvements are pretty cool too. Even though, I don't really understand that one : "The IN-early-out optimization: When doing a look-up on a multi-column index and an IN operator is used on a column other than the left-most column, then if no rows match against the first IN value, check to make sure there exist rows that match the columns to the right before continuing with the next IN value. ".…

Suppose your query is: SELECT * FROM tab WHERE key1=1 AND key2 IN (2,3,4,5); SQLite starts by doing a single b-tree lookup on the index on (1,2) - composed from the key1 field and the first possibility of the key2 field. If that works, then it proceeds to look up (1,3), (1,4), and (1,5). But if the (1,2) lookup fails, then it backs off and tries just (1, ) to see if that matches anything at all. If (1, ) finds any re…

That makes sense. I completely misunderstood the release note. Thanks for the clarification.

The insight is very interesting. I always thought there would be 2 separate binary searches...

Re: SQLite Release 3.25.0 adds support for window functions

#84

A bit off topic, but would be great to use SQLite in the browser instead of IndexedDB. I love relational databases, but you're almost forced into a NoSQL approach when developing a SPA since the client (browser) only supports simple key -> value storage. It would be a dream to use LINQ-to-SQL, or similar type safe query DSLs like Slick or Quill (Scala), or Esqueleto (Haskell) in the browser. Combine that with a singl…

https://github.com/kripken/sql.js/

I have used this and it is slow. But it was interesting!

Re: SQLite Release 3.25.0 adds support for window functions

#85
post #5

https://www.windowfunctions.com is a good introduction to window functions. Besides that, the comprehensive testing and evaluation of SQLite never ceases to amaze me. I'm usually hesitant to call software development "engineering", but SQLite is definitely well-engineered.

This looks great, but I couldn't get through the first question on aggregate functions. Are there any SQL books/tutorials that go over things like this? A lot of material I've seen has been like the classic image of "How to draw an owl. First draw two circles, then draw the rest of the owl", where they tell you the super basic stuff, then assume you know everything.

Thank you everyone for your help!

Re: SQLite Release 3.25.0 adds support for window functions

#86

A bit off topic, but would be great to use SQLite in the browser instead of IndexedDB. I love relational databases, but you're almost forced into a NoSQL approach when developing a SPA since the client (browser) only supports simple key -> value storage. It would be a dream to use LINQ-to-SQL, or similar type safe query DSLs like Slick or Quill (Scala), or Esqueleto (Haskell) in the browser. Combine that with a singl…

> A bit off topic, but would be great to use SQLite in the browser instead of IndexedDB

That almost happened. There was a thing called WebSQL [1] that was W3C was working on to add SQL to the browser. Everyone who implemented it used SQLite.

Apparently, that disqualified it from standardization. To move ahead, they wanted to see independent implementations of the standard.

No browser makers stepped up to reduce the quality of their implementation by replacing some of the best designed, best written, best tested code on the planet with some other SQL back end to satisfy the committee, and so Mozilla was able to push IndexedDB as the standard browser DB interface.

[1] https://en.wikipedia.org/wiki/Web_SQL_Database

Re: SQLite Release 3.25.0 adds support for window functions

#87

A bit off topic, but would be great to use SQLite in the browser instead of IndexedDB. I love relational databases, but you're almost forced into a NoSQL approach when developing a SPA since the client (browser) only supports simple key -> value storage. It would be a dream to use LINQ-to-SQL, or similar type safe query DSLs like Slick or Quill (Scala), or Esqueleto (Haskell) in the browser. Combine that with a singl…

This almost happened. Ten years ago "WebSQL" was being considered as a standard - it would have introduced SQLite-backed databases as a built-in browser feature: https://en.wikipedia.org/wiki/Web_SQL_Database

The W3C dropped it in November 2010 because they were uncomfortable supporting a web standard which was entirely tied to a single implementation of SQL (SQLite). I adore SQLite but I think the W3C made the right call there - web standards need more than one compatible implementation.

Re: SQLite Release 3.25.0 adds support for window functions

#88

Unrelated to window functions, but I finally took the time to start digging into SQLite's internals. People always sing its praises, so it was time to see what all the fuss was about. Someone else already mentioned that the vast majority of SQLite's codebase are tests. Well, on top of that, of the real "working" codebase I'd say the majority of it is comments. It's incredible. The source is more book than code. If yo…

> It is filled to the brim with backwards compatibility hacks that make the code more complex than it strictly needs to be

They finally ended sqlite4 as an architectural misadventure, but perhaps they should try again with the current codebase/features but all of the backwards cruft jettisoned.

Re: SQLite Release 3.25.0 adds support for window functions

#89
post #5

https://www.windowfunctions.com is a good introduction to window functions. Besides that, the comprehensive testing and evaluation of SQLite never ceases to amaze me. I'm usually hesitant to call software development "engineering", but SQLite is definitely well-engineered.

> but SQLite is definitely well-engineered.

Considering it was written for the U.S. Navy to run software on guided missile destroyers, that makes sense.

Re: SQLite Release 3.25.0 adds support for window functions

#90
post #5

https://www.windowfunctions.com is a good introduction to window functions. Besides that, the comprehensive testing and evaluation of SQLite never ceases to amaze me. I'm usually hesitant to call software development "engineering", but SQLite is definitely well-engineered.

This looks great, but I couldn't get through the first question on aggregate functions. Are there any SQL books/tutorials that go over things like this? A lot of material I've seen has been like the classic image of "How to draw an owl. First draw two circles, then draw the rest of the owl", where they tell you the super basic stuff, then assume you know everything.

The discussion at https://news.ycombinator.com/item?id=17231349 has many more links and descriptions of how to use window functions in practice. (Including my comment at the top about how in practice I find that you usually want to combine window functions with nested queries.)
Post reply on HN