With no sense of overstatement here, SQLite is one of my favorite creations in the entire world, so I have a bunch of links some of you might find interesting if you want to dig further: https://github.com/sql-js/sql.js - SQL.js lets you run SQLite within a Web page as it's just SQLite compiled to JS with Emscripten. https://litestream.io/blog/why-i-built-litestream/ - Litestream is a SQLite-powered streaming replica…
SQLite is not a toy database
291–300 of 364 posts
Re: SQLite is not a toy database
#292Earlier quoted context omitted.
How do you ensure data is not lost to oblivion if a catastrophic system failure occurs?
You put in-place a loss mitigation strategy. This strategy will vary by application. In my case, I have a similar setup where we write 25-30k records to SQLite daily. We start each day fresh with a new SQLite db file (named yyyy-mm-dd.db) and back it up to AWS S3 daily under the scheme /app_name/data/year/month/file. You could say that's 9 million records a year or 365 mini-sqlite dbs containing 25-30k records. Porta…
Re: SQLite is not a toy database
#293Earlier quoted context omitted.
SQLite is very limited because of its threading model, imo it's not usable outside of the single app model where you have a single user. https://sqlite.org/threadsafe.html https://sqlite.org/lockingv3.html
The article addresses this. Basically, you can have any number of concurrent readers , but only a single writer . Writing and reading can happen concurrently just fine. So the question is -- how many users does a website need before having only a single concurrent writer becomes a bottleneck? That number will obviously depend on the read/write ratio of any given website; but it's hard to imagine any website where [ED…
Only if your readers and writers are cleanly segregated.
Most languages and web frameworks don't have SQLite drivers out of box (or have extremely bad ones). Unlike SQLite, most databases don't really care about distinction between read-only and writable connections. So there is a good chance, that you will always open writable connection by default, because this is what your framework/ORM does. Furthermore, seemingly read-only web middleware often ends up writing to database on each request for one reason or another. If you try to reuse/pool connections (which is also important under high load), you need to be wary of keeping open writable connections in cache — again, something that does not matter to all major databases other than SQLite.
I was involved in maintenance of a small web app (db size We ended up briefly caching results of most database queries in memory, which removed most of load from database (we also did a lot of other optimizations, but this was the decisive one). Eventually the app was able to withstand up to 9000 requests per second, but none of that was an achievement of SQLite — we just evaded database, Django and Python altogether on majority of requests.
While we are on this topic, the most widespread OS in the world, Android, also has extremely low-quality SQLite drivers — despite shipping SQLite as default database for many years. Android has a broken-by-design Cursor implementation (the devs admitted it themselves [1]), that always tries to count query results, even if you don't call getCount(). And a broken connection cache, that does not support read-only connections [2] (that method used to have a "TODO", but eventually they forgot, why they wanted it, so they removed it).
1: https://medium.com/androiddevelopers/large-database-queries-...
2: https://android.googlesource.com/platform/frameworks/base/+/...
Re: SQLite is not a toy database
#294I wish SQLite had a PostgreSQL compatibility layer. Right now, to add SQLite support to a Go project (or anything without an ORM), you have to rework all your queries and migrations. It's probably an impossible ask, but having a compatibility flag within SQLite so it would accept PostgreSQL formatted queries would be extremely helpful.
Combine with a Golang AST lib and you might be able to make a CLI translate tool like `fix`.
Re: SQLite is not a toy database
#295Earlier quoted context omitted.
> SQLite doesn't really enforce column types[0], the choice is really puzzling to me. This is acknowledged as a likely mistake, but one that will never be fixed due to backward compatibility: > Flexible typing is considered a feature of SQLite, not a bug. Nevertheless, we recognize that this feature does sometimes cause confusion and pain for developers who are acustomed to working with other databases that are more…
SQLite already has different operating modes, right? e.g. WAL is turned on and stays on, I think; It seems like you could at least make type-checking an opt-in mode
Re: SQLite is not a toy database
#296Earlier quoted context omitted.
The article addresses this. Basically, you can have any number of concurrent readers , but only a single writer . Writing and reading can happen concurrently just fine. So the question is -- how many users does a website need before having only a single concurrent writer becomes a bottleneck? That number will obviously depend on the read/write ratio of any given website; but it's hard to imagine any website where [ED…
> you can have any number of concurrent readers, but only a single writer Only if your readers and writers are cleanly segregated. Most languages and web frameworks don't have SQLite drivers out of box (or have extremely bad ones). Unlike SQLite, most databases don't really care about distinction between read-only and writable connections. So there is a good chance, that you will always open writable connection by de…
Also, for your small app were you using WAL mode? Multiple readers only works in WAL journaling mode.
Re: SQLite is not a toy database
#297Earlier quoted context omitted.
> you can have any number of concurrent readers, but only a single writer Only if your readers and writers are cleanly segregated. Most languages and web frameworks don't have SQLite drivers out of box (or have extremely bad ones). Unlike SQLite, most databases don't really care about distinction between read-only and writable connections. So there is a good chance, that you will always open writable connection by de…
SQLite doesn’t require you to know which connections are read and which are write. Connections get promoted to a write lock when you do a write DDL or if you begin an IMMEDIATE transaction. Also, for your small app were you using WAL mode? Multiple readers only works in WAL journaling mode.
Our Django setup needs multiple processes to work around the Grand Interpreter Lock. Disabling connection reuse in Django config slightly changed the behavior we observed, but didn't solve the performance problem.
Re: SQLite is not a toy database
#298Earlier quoted context omitted.
I thought the same, but maybe it's useful if you start with SQLite and decide to scale up to a distributed RDBMS without having to rewrite too much?
Two thoughts: SQLAlchemy, at least, lets you switch from SQLite to Postgres in a configuration file. Isn't there a useful SQL subset which allows you to switch from one database to another without rewriting? There seems to be such a subset for C, for example, which multiple compilers all interpret the same way, and SQL is a standardized language, too.
Only if you're happy throwing away a lot of the features, which is taking away from what makes SQL an attractive solution in the first place. Plus, even some basic things aren't the same between different implementations (e.g., SELECT TOP 1 * FROM t vs SELECT * FROM t LIMIT 1), so you'd really have to be testing with different RDBMSes the whole way through to make sure you didn't accidentally break compatibility.
Re: SQLite is not a toy database
#299Earlier quoted context omitted.
Agreed, but it’s not only a question of scaling. Any app with high uptime requirements will need more than a single process from day 1.
It depends on what you consider high uptime. You can achieve 99.95% uptime with 4h of downtime a year. A lot of downtime occurs because of overly complicated systems so running a single process on a single server can give you relatively high uptime.
Re: SQLite is not a toy database
#300I was a fan of SQLite until I needed a full outer join of two tables. It was such a pain to create a workaround when I found out it isn't supported, and the result was still sub-optimal and needlessly complex. Is running a full outer joins such a unreasonable assumption within a DBMS?
I hear you. MySQL still doesn't have full outer joins either. I would think this would be one of the easier features to add. Of course, I'm sure the people working on MySQL or SQLite would say "Patches welcome!"