Oh hi, author here. Fun to see this make it to HN.
Suggestion: post this on https://sqlite.org/forum/forum - the SQLite team monitor that forum closely and I've had some really great answers from them to questions or suggestions in the past.
SQLite should have (Rust-style) editions
11–20 of 186 posts
Re: SQLite should have (Rust-style) editions
#12Oh hi, author here. Fun to see this make it to HN.
Suggestion: post this on https://sqlite.org/forum/forum - the SQLite team monitor that forum closely and I've had some really great answers from them to questions or suggestions in the past.
I haven't really ever participated in that forum before, but it's an interesting idea. I don't know if I'm going to do it, I might. Though I also wouldn't mind if someone else posted about it there. I might even make a forum account and participate in the discussion.
Re: SQLite should have (Rust-style) editions
#13SQLite is slightly different from Rust in that it is a data container. It’s somewhat more common for people to move SQLite database files from one machine to another and then inspect using the command line tool. And it is often the case that the embedded SQLite version in your app is a newer version than whatever version /usr/bin/sqlite3 happens to be. Adding editions to your SQLite file will probably break this use…
It seems SQLite could be evolve to solve this by just bundling itself entirely in the data files? After all, the binary is less than 1MB, anywhere you’re putting a database surely has at least that much overhead, for most applications it’s less than a drop in the bucket.
I’d be interested to learn if there are any db implementations that take this approach, or reasons this wouldn’t work.
Re: SQLite should have (Rust-style) editions
#14SQLite is slightly different from Rust in that it is a data container. It’s somewhat more common for people to move SQLite database files from one machine to another and then inspect using the command line tool. And it is often the case that the embedded SQLite version in your app is a newer version than whatever version /usr/bin/sqlite3 happens to be. Adding editions to your SQLite file will probably break this use…
editions could be self-describing as to certain semantic changes, and you could embed that with the file. older versions could safely ignore it and newer versions parse and run it. you could also force things like "editions must be declared early", "editions are one way only" etc to get some level of security in the adoption of the change
Re: SQLite should have (Rust-style) editions
#15SQLite is slightly different from Rust in that it is a data container. It’s somewhat more common for people to move SQLite database files from one machine to another and then inspect using the command line tool. And it is often the case that the embedded SQLite version in your app is a newer version than whatever version /usr/bin/sqlite3 happens to be. Adding editions to your SQLite file will probably break this use…
It seems SQLite could be evolve to solve this by just bundling itself entirely in the data files? After all, the binary is less than 1MB, anywhere you’re putting a database surely has at least that much overhead, for most applications it’s less than a drop in the bucket. I’d be interested to learn if there are any db implementations that take this approach, or reasons this wouldn’t work.
Well you'd have the problem that an sqlite database file created on a Linux AMD64 box could be copied to an AArch64 macOS machine to be read there. And quite a lot of (cross platform) software build their file formats on top of SQLite.
Re: SQLite should have (Rust-style) editions
#16Earlier quoted context omitted.
For some of these pragmas you have the same issue with or without "editions", right? Busy timeout is per-connection. And then: if you're running in WAL mode, you, the user, have to know that, or risk messing up the database by copying just the .db file rather than vacuuming-into.
Editions make the problem worse by requiring the version not only to support the underlying pragmas but also to understand the edition mapping. Example: PRAGMA foo=1 is introduced in 2027. PRAGMA edition=2030 implies this foo pragma. Now you unnecessarily lock out three years worth of releases.
I don't see how you don't have the pragma compatibility problem either way. The edition proposal captures a bunch of behaviors that already exist.
Re: SQLite should have (Rust-style) editions
#17In the first example, there's a a second thing that surprised me: you delete an entity and it's unique ID gets reused? Is that a good idea?
I guess if foreign keys are handled properly then that's not a problem by definition? But it sounds wrong somehow.
Re: SQLite should have (Rust-style) editions
#18Earlier quoted context omitted.
Editions make the problem worse by requiring the version not only to support the underlying pragmas but also to understand the edition mapping. Example: PRAGMA foo=1 is introduced in 2027. PRAGMA edition=2030 implies this foo pragma. Now you unnecessarily lock out three years worth of releases.
I don't see how you don't have the pragma compatibility problem either way. The edition proposal captures a bunch of behaviors that already exist.
Yes, the problem exists either way. Editions just exacerbate the problem.
Re: SQLite should have (Rust-style) editions
#19It might be worth bringing this up on the forum [1]. The developers are quite active there, and it's possible they've never considered this option, or they have considered it and have reasons to not go for it. The original design followed Postel's Law (see my comment from the other say [2]), it would (theoretically) be nice if that mess could be avoided by specifying an edition. Today I noticed I could do `pragma for…
The Postfix mailer has allowed recommended default behavior to evolve using its "compatibility_level" parameter:
https://www.postfix.org/postconf.5.html#compatibility_level
https://www.postfix.org/COMPATIBILITY_README.html
You get a warning whenever you depend on the deprecated old default until you either move forward or specifically commit to the old behavior.
Re: SQLite should have (Rust-style) editions
#20Earlier quoted context omitted.
It seems SQLite could be evolve to solve this by just bundling itself entirely in the data files? After all, the binary is less than 1MB, anywhere you’re putting a database surely has at least that much overhead, for most applications it’s less than a drop in the bucket. I’d be interested to learn if there are any db implementations that take this approach, or reasons this wouldn’t work.
Well you'd have the problem that an sqlite database file created on a Linux AMD64 box could be copied to an AArch64 macOS machine to be read there. And quite a lot of (cross platform) software build their file formats on top of SQLite.
I think my silly and unserious naive response would be linear scaling bundled binaries with number of platforms doesn’t seem to be that materially different in terms of total size. But I see your point, didn’t consider the architectures