Live data from Hacker News

SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

micrologics.org

31–40 of 87 posts

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#31
post #19

Earlier quoted context omitted.

> Better option is to just add a new column with the correct definition After that you won't be able to change column to NOT NULL. You would need migration to create new table with not null column, copy everything, drop old table and rename the new one. Edit: unless the table is empty.

Wrong, you can change NOT NULL since 3.53: https://sqlite.org/releaselog/3_53_3.html

Released a month ago, thanks, didn't know.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#32
Unfortunately written by an AI - that completely takes the wind out of the content and makes me not even want to read any further. The distinction between “production” and “non-production” is also questionable. For an evaluation, I recommend the following original articles (certainly not AI-generated):

- https://sqlite.org/whentouse.html

- https://sqlite.org/different.html

- https://sqlite.org/quirks.html

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#33
post #23
post #19

Earlier quoted context omitted.

> Better option is to just add a new column with the correct definition After that you won't be able to change column to NOT NULL. You would need migration to create new table with not null column, copy everything, drop old table and rename the new one. Edit: unless the table is empty.

How do you migrate in place data that doesn’t convert between types while maintaining a strict condition like NOT NULL? This is again a scenario I’ve never run into 20ish years of SQL.

You create a nullable column and then change it to not null. Which wasn't possible in SQLite until recently.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#34
post #3

I'm fairly confident this is AI generated, but it makes me think regardless: Whenever I see these kind of articles, I'm left wondering if they've actually used SQLite in production because I always see points about how to optimize performance, like using the WAL, but never about annoyances/issues you'd run into before even needing to worry about that. I guess it's the zeitgeist to use it in a production setting, and…

SQLite supports ALTER TABLE:

https://www.sqlite.org/lang_altertable.html

On Go you can embed your migrations in your binary:

https://oscarforner.com/blog/2023-10-10-go-embed-for-migrati...

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#35

I love SQLite, and I really want to run it in production, but my clients expect minimal data loss and downtime when one of my servers goes down. The answer to that being running it on top of LiteFS or LiteStream seems like starts to make the setup a lot less simple and lot less battle tested, which kind of starts to negate the advantages over just running Postgres.

all clients think they're google

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#36

I love SQLite, and I really want to run it in production, but my clients expect minimal data loss and downtime when one of my servers goes down. The answer to that being running it on top of LiteFS or LiteStream seems like starts to make the setup a lot less simple and lot less battle tested, which kind of starts to negate the advantages over just running Postgres.

all clients think they're google

I dunno, not losing data and minimizing unnecessary downtime is table-stakes for a pretty significant class of businesses.

If for no other reason, dealing with these at any scale often distracts from running your actual business.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#37
post #10

As someone really tempted to use SQLite in production, the one thing I keep bumping against is how to have a nice GUI to interact with the running database. With our current prod databases, I can connect dbeaver and the like to them and nicely browse the data, query, or even do the occasional fix. Seems like this would be much more of a head scratcher if the database is just a file on the same VPS the app runs on.

sqlite-web, put it behind nginx with basic auth or configure a password.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#38
post #33
post #23

Earlier quoted context omitted.

How do you migrate in place data that doesn’t convert between types while maintaining a strict condition like NOT NULL? This is again a scenario I’ve never run into 20ish years of SQL.

You create a nullable column and then change it to not null. Which wasn't possible in SQLite until recently.

I’m not sure what your original point is pointing out.

Some data doesn’t convert is what I’m pointing out regardless of Postgres or SQLite.

Re: SQLite in Production: Optimizing WAL Mode, Concurrency, and VFS Layers

#40
post #3

I'm fairly confident this is AI generated, but it makes me think regardless: Whenever I see these kind of articles, I'm left wondering if they've actually used SQLite in production because I always see points about how to optimize performance, like using the WAL, but never about annoyances/issues you'd run into before even needing to worry about that. I guess it's the zeitgeist to use it in a production setting, and…

My sqlite-utils CLI tool and Python library offers solutions to both the alter table limitations and the need for schema migrations.

For alter table it offers a "transform" command which implements the pattern of creating a new table with your desired scheme, copying data to it from the old table, then renaming the tables (all in a transaction): https://sqlite-utils.datasette.io/en/stable/cli.html#transfo...

  sqlite-utils transform fixtures.db roadside_attractions \
    --rename pk id \
    --default name Untitled \
    --column-order id \
    --column-order longitude \
    --column-order latitude \
    --drop address
And for migrations there's a new-in-v4 "migrate" command which lets you create and execute an ordered sequence of migrations: https://sqlite-utils.datasette.io/en/stable/cli.html#running...

  sqlite-utils migrate creatures.db path/to/migrations.py
Migrations files look like this: https://sqlite-utils.datasette.io/en/stable/migrations.html#...

  from sqlite_utils import Migrations
  
  migrations = Migrations("creatures")
  
  @migrations()
  def create_table(db):
      db["creatures"].create(
          {"id": int, "name": str, "species": str},
          pk="id",
      )
  
  @migrations()
  def add_weight(db):
      db["creatures"].add_column("weight", float)
Post reply on HN