Live data from Hacker News

SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

micrologics.org

41–50 of 87 posts

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#41
post #3

I'm fairly confident this is AI generated, but it makes me think regardless: Whenever I see these kind of articles, I'm left wondering if they've actually used SQLite in production because I always see points about how to optimize performance, like using the WAL, but never about annoyances/issues you'd run into before even needing to worry about that. I guess it's the zeitgeist to use it in a production setting, and…

A handy trick for column types is check constraints.

You can define constraints on a column that ensure it is text that's valid JSON for example:

  CREATE TABLE documents (
    id INTEGER PRIMARY KEY,
    data TEXT NOT NULL
      CHECK (
        json_valid(data)
        json_type(data) = 'object'
      )
  );
Or to ensure specific keys:

  CREATE TABLE documents (
    id INTEGER PRIMARY KEY,
    data TEXT NOT NULL CHECK (
      json_valid(data)
      AND json_type(data) = 'object'
      AND json_type(data, '$.name') = 'text'
      AND json_type(data, '$.age') = 'integer'
    )
  );
You can even use this for things like enforcing a valid YYYY-MM-DD date, though that gets a bit convoluted:

  CREATE TABLE events (
    id INTEGER PRIMARY KEY,
    occurred_on TEXT NOT NULL CHECK (
      length(occurred_on) = 10
      AND occurred_on GLOB
        '[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'
      AND date(occurred_on, '+0 days') = occurred_on
    )
  );

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#42
post #3

I'm fairly confident this is AI generated, but it makes me think regardless: Whenever I see these kind of articles, I'm left wondering if they've actually used SQLite in production because I always see points about how to optimize performance, like using the WAL, but never about annoyances/issues you'd run into before even needing to worry about that. I guess it's the zeitgeist to use it in a production setting, and…

> - You have limited options for dealing with schema migrations.

That's the biggest pain of dealign with SQLite.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#43

Earlier quoted context omitted.

all clients think they're google

I dunno, not losing data and minimizing unnecessary downtime is table-stakes for a pretty significant class of businesses. If for no other reason, dealing with these at any scale often distracts from running your actual business.

Sqlite with litestream has a smaller default window of data loss than RDF. 1s vs 5 minutes.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#44

Unfortunately written by an AI - that completely takes the wind out of the content and makes me not even want to read any further. The distinction between “production” and “non-production” is also questionable. For an evaluation, I recommend the following original articles (certainly not AI-generated): - https://sqlite.org/whentouse.html - https://sqlite.org/different.html - https://sqlite.org/quirks.html

[flagged]

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#45
post #40
post #3

I'm fairly confident this is AI generated, but it makes me think regardless: Whenever I see these kind of articles, I'm left wondering if they've actually used SQLite in production because I always see points about how to optimize performance, like using the WAL, but never about annoyances/issues you'd run into before even needing to worry about that. I guess it's the zeitgeist to use it in a production setting, and…

My sqlite-utils CLI tool and Python library offers solutions to both the alter table limitations and the need for schema migrations. For alter table it offers a "transform" command which implements the pattern of creating a new table with your desired scheme, copying data to it from the old table, then renaming the tables (all in a transaction): https://sqlite-utils.datasette.io/en/stable/cli.html#transfo... sqlite-u…

I'd def recommend simonw's CLI and Python lib for anyone doing SQLite stuff, there's all kinds of stuff there.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#47
Where I part ways a little with the recommendation is in the busy_timeout + BEGIN IMMEDIATE suggestion. On an embedded system, that's not really sufficient; I maintain a caching service that is used by thousands of clients. It ingests data over MQTT and has a web interface for queries. In my design, there is a MQTT thread and pruning thread. The MQTT ingestion thread and the pruning thread can both end up "contending" for the write lock, and increasing the busy_timeout only makes them wait longer before noticing that contention and moving on to the next task. What we ended up needing to do was add an application-level lock that only allows a single writer transaction to be in flight at a time; BEGIN IMMEDIATE is nice for preventing other readers from blocking, but it doesn't help much with other writers.

Meanwhile, you might be surprised how often “single-tenant edge deployment” comes up in embedded contexts; a Pi with an SD card is a common configuration for an embedded database, and it's right at the intersection of these problems.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#49
> PRAGMA synchronous = NORMAL;

> In NORMAL mode, the database engine syncs to disk only at critical moments (e.g., during checkpoints) rather than at every single transaction commit. In WAL mode, this is completely safe from database corruption; even if the server crashes, only the uncommitted transactions in the WAL are lost, but the database integrity remains intact.

No, this is not safe. You can lose the latest committed transaction with this pragma.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#50
post #21

I am obsessed with the idea of per tenant databases. But I am afraid of migrations. Has anyone tried that?

We do it on Postgres with schemas but we have a very fixed amount of tenants so it works. I don't think it scales when you have unbounded amounts of tenants.

It works well enough if you deploy a separate version of the app for each tenant, and upgrade each tenant's app + db in lockstep.

It's a bit of a 1990s setup (as the sibling comment says, it's more or less the same model as a widely-deployed desktop app), but it does scale and can be useful in situations where strict isolation between tenants is beneficial.

Post reply on HN