Live data from Hacker News

SQLite in Production: Lessons from Running a Store on a Single File

ultrathink.art

101–110 of 135 posts

Re: SQLite in Production: Lessons from Running a Store on a Single File

#101

I see tons of articles like this, and I have no doubt sqlite proved to be a great piece of software in production environments, but what I rarely find discussed is that we lack tools that enable you to access and _maintain_ SQLite databases. It's so convenient to just open Datagrip and have a look at all my PostgreSQL instances; that's not possible with sqlite AFAIK (not even SSH tunnelling?). If something goes wrong…

I have a project to help with that:

  uvx datasette data.db
That starts a web app on port 8001 that looks like this:

https://latest.datasette.io/fixtures

Re: SQLite in Production: Lessons from Running a Store on a Single File

#102
post #99

Earlier quoted context omitted.

Ooh new historical Unix variant I had never heard of.. neat!

AIX is still supported and sold, so quite current? Some that I used that are gone... Ultrix (MIPS), Clix, Irix, SunOS 4, SCO OpenServer, TI System V. https://en.wikipedia.org/wiki/Ultrix https://en.wikipedia.org/wiki/Intergraph

NeXTstep? (Leaving aside fun spitballing about whether Tahoe is morally OPENSTEP 26, and whether it was NeXT that actually bought Apple for negative $400 million...)

Re: SQLite in Production: Lessons from Running a Store on a Single File

#103
post #63

Earlier quoted context omitted.

WAL relies on shared memory, so while a proper FS is necessary, it isn't going to help in this case.

Why does it not help if both containers can mmap the same -shm file?

Shared memory across containers is a property of a containerization environment, not a property of a file system, "proper" or not.

Re: SQLite in Production: Lessons from Running a Store on a Single File

#104
post #14

SQLite has a ".backup" command that you should always use to backup a SQLite DB. You're risking data loss/corruption using "cp" to backup your database as prescribed in the article. https://sqlite.org/cli.html#special_commands_to_sqlite3_dot_...

Yeah, using cp to backup sqlite is a very bad idea. And yet, unless you know this, this is what Claude etc will implement for you. Every friggin' time.

Well, humans also default to 'cp' until they learn the better pattern or find out their backup is missing data.

Also, my n=1 is that I told Claude to create a `make backup` task and it used .backup.

I don't understand the double standard though. Why do we pretend us humans are immaculate in these AI convos? If you had the prescience to be the guy who looked up how to properly back up an sqlite db, you'd have the prescience to get Claude to read docs. It's the same corner cut.

There's this weird contradiction where we both expect and don't expect AI to do anything well. We expect it to yolo the correct solution without docs since that's what we tried to make it do. And if it makes the error a human would make without docs, of course it did, it's just AI. Or, it shouldn't have to read docs, it's AI.

Re: SQLite in Production: Lessons from Running a Store on a Single File

#105
> The sqlite_sequence table is the most underappreciated debugging tool in SQLite. It tracks the highest auto-increment value ever assigned for each table — even if that row was subsequently lost. > WorkQueueTask.count returns ~300 (current rows). The sequence shows 3,700+ (every task ever created). If those numbers diverge unexpectedly, something deleted rows it shouldn't have.

Or it means that SQLite is exhibiting some of its "maybe I will, maybe I won't" behavior [0]:

> Note that "monotonically increasing" does not imply that the ROWID always increases by exactly one. One is the usual increment. However, if an insert fails due to (for example) a uniqueness constraint, the ROWID of the failed insertion attempt might not be reused on subsequent inserts, resulting in gaps in the ROWID sequence. AUTOINCREMENT guarantees that automatically chosen ROWIDs will be increasing but not that they will be sequential.

> No ILIKE. PostgreSQL developers reach for WHERE name ILIKE '%term%' instinctively. SQLite throws a syntax error. Use WHERE LOWER(name) LIKE '%term%' instead.

You should not be reaching for ILIKE, functions on predicates, or leading wildcards unless you're aware of the impacts those have on indexing.

> json_extract returns native types. json_extract(data, '$.id') returns an integer if the value was stored as a number. Comparing it to a string silently fails. Always CAST(json_extract(...) AS TEXT) when you need string comparison.

If you're using strings embedded in JSON as predicates, you're going to have a very bad time when you get more than a trivial number of rows in the table.

0: https://sqlite.org/autoinc.html

Re: SQLite in Production: Lessons from Running a Store on a Single File

#106
> The Fix: Stop Deploying So Fast

I don't know how Ultrathink works, and I have no "real world" experience with Kamal, but I find it intriguing to see someone consider 11 deployments in 2 hours to be "fast".

Instead of handicapping yourself, fix your deployment pipeline, 10 min deploys are not OK for an online store.

Re: SQLite in Production: Lessons from Running a Store on a Single File

#107

Earlier quoted context omitted.

"I know about the .backup command, there's no way I'm using cp to backup the SQLite db from production." Oh. Guess I know what I'm fixing before lunch. Thank you :)

Yes, especially if you are using a WAL.

Totally. It also explains why I was confused to find the WAL files when I was testing the backups last week.

Re: SQLite in Production: Lessons from Running a Store on a Single File

#108
post #99

Earlier quoted context omitted.

AIX is still supported and sold, so quite current? Some that I used that are gone... Ultrix (MIPS), Clix, Irix, SunOS 4, SCO OpenServer, TI System V. https://en.wikipedia.org/wiki/Ultrix https://en.wikipedia.org/wiki/Intergraph

NeXTstep? (Leaving aside fun spitballing about whether Tahoe is morally OPENSTEP 26, and whether it was NeXT that actually bought Apple for negative $400 million...)

Alas, I never had access to any of the Next environments, until PPC MacOS.

I did hold a copy in my hands for 486-class machines in the college bookstore.

Re: SQLite in Production: Lessons from Running a Store on a Single File

#109
post #53

Earlier quoted context omitted.

You accept downtime. That's the limitation of SQLite. Or you use some distributed SQLite tool like rqlite, etc

I'm personally fine with a little bit of downtime for my particular small app. I'm just surprised there's not a more detailed story around deploying sqlite in a high availability prod environment given it's increased popularity and coverage over the last few years. Especially surprising with Rails' (my stack) going full "sqlite-first".

The "sqlite-first" folks have accepted that a bit of downtime is better than engineering wildly complex systems that avoid it, for non-mission-critical apps (if your mission is a low volume e-commerce shop.. it's not critical)

Re: SQLite in Production: Lessons from Running a Store on a Single File

#110
post #44

> Backups are cp production.sqlite3 backup.sqlite3 I use gobackup[0] as another container in compose.yml file which can backup to multiple locations. [0]: https://gobackup.github.io/

Does cp actually work on live sqlite files? I wouldn’t expect it to, since cp does not create a crash-consistent snapshot.

Maybe if the system is idle
Post reply on HN