Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SQLite should have (Rust-style) editions

mort.coffee

Post reply on HN
Print view View original

151–160 of 186 posts

Re: SQLite should have (Rust-style) editions

#151

Post by Hendrikto »

dolmen wrote: post #145

Counterpoint: each new edition will add bloat to SQLite as future edition will need to keep each past edition pragma set for backward compatibility. As SQLite is often used embedded, bloat matters. So I suggest that "PRAGMA edition" to be only be a shortcut to a list of PRAGMA commands, that would be expanded at the library level: PRAGMA edition would never appear in the DB file. As such, the build of the library wou…

Isn’t that exactly what the author suggests? Editions as a set of config options that are already there.

The config options are already implemented, so the heavy lifting is done. Editions would amount to a few hundred bytes.

Re: SQLite should have (Rust-style) editions

#152

Post by doc_ick »

tptacek wrote: post #61
doc_ick wrote: post #40

Earlier quoted context omitted.

I’d say these are reasonable settings for most uses. Though do you know of surveys that back this up? I don’t mean to nit pick too much, I’d just like to see common uses and the data.

SQLite is used in a lot of unconventional settings (for SQL databases) where these settings don't make as much sense. But that's what makes the "edition" useful; it captures the use case we all mean when we're thinking of the "database" lego in an application stack.

So you’re saying there’s no data to group these settings by editions.

Re: SQLite should have (Rust-style) editions

#153

Post by eesmith »

andai wrote: post #22
mort96 wrote: post #15

Earlier quoted context omitted.

Well you'd have the problem that an sqlite database file created on a Linux AMD64 box could be copied to an AArch64 macOS machine to be read there. And quite a lot of (cross platform) software build their file formats on top of SQLite.

Perhaps it could be an "actually portable executable"? https://news.ycombinator.com/item?id=26273960

FWIW, that's existed for a while. Source at https://github.com/jart/cosmopolitan/tree/master/third_party... with downloadable binary at https://cosmo.zip/pub/cosmos/bin/sqlite3 .

  % curl -LO 'https://cosmo.zip/pub/cosmos/bin/sqlite3'
    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  100 4845k  100 4845k    0     0  1384k      0  0:00:03  0:00:03 --:--:-- 1384k
  % chmod +x sqlite3
  % file sqlite3
  sqlite3: DOS/MBR boot sector; partition 1 : ID=0x7f, active, start-CHS (0x0,0,1), end-CHS (0x3ff,255,63), startsector 0, 4294967295 sectors
  % ./sqlite3
  SQLite version 3.40.0 2022-11-16 12:10:08
  Enter ".help" for usage hints.
  Connected to a transient in-memory database.
  Use ".open FILENAME" to reopen on a persistent database.
  sqlite>:

Re: SQLite should have (Rust-style) editions

#154

Post by Bratmon »

dolmen wrote: post #145

Counterpoint: each new edition will add bloat to SQLite as future edition will need to keep each past edition pragma set for backward compatibility. As SQLite is often used embedded, bloat matters. So I suggest that "PRAGMA edition" to be only be a shortcut to a list of PRAGMA commands, that would be expanded at the library level: PRAGMA edition would never appear in the DB file. As such, the build of the library wou…

It's always really funny to me when commenters don't read the article and then phrase the article's point like they invented it.

Re: SQLite should have (Rust-style) editions

#155

Post by tialaramex »

uowlfinder wrote: post #137
tialaramex wrote: post #115

Earlier quoted context omitted.

> Now we need this for C/C++ P1881 Epochs proposed to WG21 (the C++ standards committee) in 2019 by Vittorio Romeo The committee found plenty of problems with this, and made it clear that if Vittorio did all the hard work to resolve those problems they would find more, P1881 was abandoned. There was a Reddit thread https://www.reddit.com/r/cpp/comments/1tja9zr/c_profiles_a_c... which suggested that the "Profiles" ide…

[dead]

> Does it make the compiler and other tools larger and more complex for each new edition?

Marginally.

> Are the migration tools always fully automatic, quick to execute, and flawless? Is any manual work, or expertise, required for upgrading in the worst case?

The ambition is always automatic migration, but there are almost invariably weird edge cases. For one thing Rust has a macro which includes arbitrary text in your program, like the C pre-processor #include but much less frequently needed, so short of auto-fixing all text documents on your computer such a migration will always be somewhat limited.

> If I read a blog about Rust, and the blog forgot to mention which edition it uses for its code examples, can that cause any problems? Should I just skip the blog if it is old?

Depending on how old it is, like anything else you read on a blog it might now be obsolete regardless of editions. Blog posts written last week about how England will face France in a World Cup final are irrelevant, both play a runners-up game instead on Saturday.

> As an example, why are these popular projects still on Rust edition 2021?

They all look like they're pretty mature, so, probably nobody thought it was worth doing?

Re: SQLite should have (Rust-style) editions

#156

Post by jmull »

I actually think this wouldn’t be that helpful.

The developer still needs to ensure they apply their set of pragmas, whether that’s a single edition pragma or a set of pragmas. And they still need to understand and carefully choose the pragmas/options they use (an edition really makes this a little harder by abstracting/hiding something that needs to be directly understood and visible).

And these proposed new default pragmas are more incremental improvements rather than complete solutions, more convenience than critical. E.g., while the default affinity is goofy, strict tables don’t come close to a comprehensive validation mechanism — so if you need strong validation you’re likely going to need to implement that at a higher level anyway. (Also, there’s a decent separation-of-concerns argument that you should handle it separately.)

Likewise the busy timeout. The pragma is convenient but you still need to handle busy timeouts. (The author’s problem, “I didn’t realize busy timeouts could happen so the app didn’t handle them correctly”, is not solved by the pragma.)

Re: SQLite should have (Rust-style) editions

#157

Post by platz »

tzot wrote: post #149
platz wrote: post #97

Earlier quoted context omitted.

busy_timeout is often sidestepped (ignored) when a transaction attempts to upgrade from a read to a write producing SQLITE_BUSY. By default, SQLite transactions start in DEFERRED mode, acting as read transactions until an actual write operation occurs. If another connection begins writing to the database while your transaction is in this read state, an immediate SQLITE_BUSY error is triggered regardless of what you s…

Exactly. In cases where I expect long-running parallel connections from separate processes to the same sqlite file, I make sure that all read transactions do `BEGIN DEFERRED` so `COMMIT` releases the read locks, and all write transactions do `BEGIN IMMEDIATE` so that `SQLITE_BUSY` timeout is not side-stepped. There was one case where all transactions were implemented using nested `SAVEPOINT bla` so `BEGIN IMMEDIATE`…

lol, I briefly thought of such a technique, but in the end, found it simpler to just use a synchronization primitive at the application level to serialize db access on transactions where I knew I was going to write. it amounts to about the same honor code.

Re: SQLite should have (Rust-style) editions

#158

Post by mort96 »

mort96 wrote: post #138
Certhas wrote: post #120

Earlier quoted context omitted.

That's the idea behind the future file format: https://github.com/future-file-format/F3 Bundle a wasm decoder as a fallback when native decoder isn't available.

The issue then is that instead of just bundling a tiny 1MB sqlite library with your program, you have to bundle a huge WASM runtime. I looked at the size of wasm on my system just to get an idea and it's 46MB; that's huge . For context, I have a game I'm working on where I build stand-alone builds with all dependencies bundled. The app bundle is 14MB right now. If I added sqlite, it would grow to 15MB. If I added sql…

Ugh I can't edit now but just noticed I should've said I looked at the size of wasmtime, not "the size of wasm".

Re: SQLite should have (Rust-style) editions

#159

Post by uowlfinder »

tialaramex wrote: post #155
uowlfinder wrote: post #137

Earlier quoted context omitted.

[dead]

> Does it make the compiler and other tools larger and more complex for each new edition? Marginally. > Are the migration tools always fully automatic, quick to execute, and flawless? Is any manual work, or expertise, required for upgrading in the worst case? The ambition is always automatic migration, but there are almost invariably weird edge cases. For one thing Rust has a macro which includes arbitrary text in yo…

Reading between the lines, it is as if you consider there to be large problems with Rust editions, but you do not wish to make Rust, Rust editions, nor the programming language concept of editions, look bad.

Does upgrading a large project require intimate knowledge of that project in the worst case? How much work is it in the worst case?

How do Rust editions and macros interact? Does definition of macros or usage of macros make upgrading a Rust edition harder? If yes, how much harder in the worst case?

Re: SQLite should have (Rust-style) editions

#160

Post by Rygian »

ncruces wrote: post #143
Rygian wrote: post #114

Earlier quoted context omitted.

The section "The solution: editions?" in the article addresses directly the point of existing data. The way I read it, this article does not advocate at any point to change the defaults for existing databases, but rather to start with better defaults for new databases. Also, regarding the timeout of 5 seconds, I disagree with your premise "SQLite doesn't decide, which makes perfect sense". As the article explains, SQ…

> The section "The solution: editions?" in the article addresses directly the point of existing data. I'm sorry, where? SQLite schema is stored as text. If you change the default interpretation of CREATE TABLE with a PRAGMA, your existing tables become STRICT, but (1) they might now have columns with invalid types (which means you have an invalid schema, and your database fails to open), (2) they may have invalid dat…

> > The section "The solution: editions?" in the article addresses directly the point of existing data.

> I'm sorry, where?

Here:

quote

[...]This should be a nice middle ground which avoids breaking backwards compatibility, but lets the database engine move forwards and not be bogged down by its own history.

end quote

> Regarding busy_timeout, why is 5s specifically a better default?

According to the post we are discussing, any number greater than zero is better than the current default:

quote

The default behavior [no timeout] has lead me to writing real-world bugs, where systems would sometimes just crash. I've manually written retry loops to fix it.

end quote

Your forum links may be quite useful as a response to this comment: https://news.ycombinator.com/item?id=48928441

Post reply on HN