Live data from Hacker News

SQLite is not a toy database

antonz.org

131–140 of 364 posts

Re: SQLite is not a toy database

#131
post #28

Earlier quoted context omitted.

That's not the case no, you usually access your database with thread pool. Otherwise everyone would wait until the single connection is free. Once you have a bit more users that tries to write everything will fall appart.

Well, let me throw out a crazy idea: Nobody said you have to use a single database/file. Obviously, you are going to want to spend a couple minutes thinking about referential integrity. But how often do you delete records in your web app?

As an IRC client with an SQLite backend: several hundred times per second.

Per user.

While reads are more like once every day per user.

Re: SQLite is not a toy database

#132
post #108

With no sense of overstatement here, SQLite is one of my favorite creations in the entire world, so I have a bunch of links some of you might find interesting if you want to dig further: https://github.com/sql-js/sql.js - SQL.js lets you run SQLite within a Web page as it's just SQLite compiled to JS with Emscripten. https://litestream.io/blog/why-i-built-litestream/ - Litestream is a SQLite-powered streaming replica…

Wow! The sql.js bundle is only 8kb. This is a hidden gem for sure. Thanks for pointing it out! EDIT: It's actually 1.2MB. Thanks for pointing it out :)

[deleted]

Re: SQLite is not a toy database

#133
post #84

PostgreSQL is not a toy database. MySQL is not a toy database. I feel like the "toy database" part is clickbait

Not sure about that. I can't recall having seen PostgreSQL being referred to as toy database. I have seen however texts cautioning SQLite in production use, e.g. in [1], [2].

[1] https://docs.bareos.org/bareos-18.2/IntroductionAndTutorial/...

[2] https://docs.bareos.org/bareos-18.2/DeveloperGuide/catalog.h...

Re: SQLite is not a toy database

#134

Earlier quoted context omitted.

How do backups work? Is there a locking mechanism to prevent file corruption if the file is copied during writing?

It supports multiple readers, and writes are atomic - no reader should ever get corrupt data just because there is a write happening in parallel. You probably should not use the OS to copy the live db file, rather, have an in-process thread that reads the db and writes to a backup location periodically.

Exactly, many forget the little detail that they should use the backup feature, rather than just copy the database file.

Re: SQLite is not a toy database

#135
post #16

> There is a popular opinion among developers that SQLite is not suitable for the web, because it doesn’t support concurrent access. No, the issue is it doesn't have high availability features: failover, snapshots, concurrent backups, etc. (Edit: oops, comment pointed out it does have concurrent backups.) SQLite isn't a toy DBMS, it's an extremely capable embedded DBMS. An embedded DBMS is geared towards serving a si…

I agree that high availability features are outside of the goals of an embedded database. There's an ecosystem of tools SQLite to provide these benefits though. There's dqlite & rqlite for providing HA over SQLite. I'm the author of Litestream[1] which provides streaming replication to S3 for SQLite databases so folks can safely run a single node instance. [1]: https://litestream.io/

One could, for instance, run “static” sites with the data stored in a SQLite database. Update on write apps often can do okay with 90% uptime. It’s the “generate everything in every request” crowd that needs an HA solution and why do we keep doing this to ourselves?

Re: SQLite is not a toy database

#136

Earlier quoted context omitted.

Does that mean it's okay for your application to loose transactions (which occured between the backup point and the failure point) or do you have other mitigations ?

Guess this daily 2 seconds of downtime is worth it, when that reduces cost say from $2000/month to $20/month.

Many banks still “shutdown” for hours every night to do backups.

Re: SQLite is not a toy database

#137
post #86

Earlier quoted context omitted.

Huh? SQLite is public domain.[0] There are paid extensions for encrypted and compressed databases[1], but those cost a flat fee for unlimited devices:[2] "Your license is perpetual. You have paid a one-time fee that allows you to use and modify the software forever. You can ship as many copied of the software to your customers as you want so long as you ensure that only compiled binaries are shipped (you cannot distr…

Interesting - it's been a long time since I used / looked at it. There was a big uproar in the company though and a drive to move away from it due to licensing and cost from there. Maybe they where misinformed.

I would say terribly so.

Re: SQLite is not a toy database

#138
post #16

> There is a popular opinion among developers that SQLite is not suitable for the web, because it doesn’t support concurrent access. No, the issue is it doesn't have high availability features: failover, snapshots, concurrent backups, etc. (Edit: oops, comment pointed out it does have concurrent backups.) SQLite isn't a toy DBMS, it's an extremely capable embedded DBMS. An embedded DBMS is geared towards serving a si…

no, those things have never stopped me. What has stopped me is that the default configuration for a Rails app throws intermittent exceptions when I'm trying to save, because concurrent writes fail and Rails (at least as of a couple years ago) does the wrong thing by default

Re: SQLite is not a toy database

#139
post #122

Earlier quoted context omitted.

I agree that PostgreSQL has some great tooling around it. For SQLite replication, you might want to checkout Litestream: https://litestream.io/

I know about litestream, it's not the kind of "setup and forget" I have in mind. But it's nice that it exists.

That's good to know. I'm always trying to improve the developer ergonomics. What would you want to see added to make it "setup and forget"?
Post reply on HN