41–50 of 98 posts
Earlier quoted context omitted.
WAL mode makes those redundant. Just use WAL mode.
We all wish you were right. But alas life's not that simple. I suggest reading the manual on the section: "Sometimes Queries Return SQLITE_BUSY In WAL Mode" https://sqlite.org/wal.html#sometimes_queries_return_sqlite_...
I like SQLite and I like Rails but this seems synonymous with using MS Access in a production environment.
I like SQLite and I like Rails but this seems synonymous with using MS Access in a production environment.
I mean this works for Pieter Levels. Definitely will cause issues when you get above a certain threshold of users.
2015
> It’d analyze the feed, see which jobs were remote, then normalize the data and then push it into a simple SQLite database (yes, I’m not using JSON text files as a database anymore, thank you :P).
https://levels.io/how-i-build-my-minimum-viable-products/
2014
> This includes all data used by the app. I actually shy away from using real database systems in the 12 startups. Instead I use JSON text files.
I was hoping to find data on Pieter's current projects using Sqlite and their loads. (For all I know, RemoteOK still does, but I can't find more recent posts about it).
General SQLite question for the group… I’m making a FOSS analytics system, and ease-of-installation is important. I want to send event data to a separate SQLite database, to keep analytics data separate from the main app’s data. I’m concerned about scaling, since even a modestly busy website could have 1000+ events per second. My thought is to store events in memory on the server and then make one batched write every…
Write events in parquet format and use DuckDB for your analytics?
I’ve had good experience with ClickHouse, too, but it feels a bit more like Postgres rather than SQLite in terms of portability.
I like SQLite and I like Rails but this seems synonymous with using MS Access in a production environment.
General SQLite question for the group… I’m making a FOSS analytics system, and ease-of-installation is important. I want to send event data to a separate SQLite database, to keep analytics data separate from the main app’s data. I’m concerned about scaling, since even a modestly busy website could have 1000+ events per second. My thought is to store events in memory on the server and then make one batched write every…
You can write large SQLite databases at disk speed with the right data model and schema. What most people do as mentioned is either batch writes, or simply send them over some kind of channel to a single thread that is the designated writer, some kind of MPSC queue, and that queue effectively acts as a serialization barrier. Either can work depending on your latency/durability requirements. You also absolutely need t…
Another great pattern for batch writes is to use a system like Nagle's algorithm - push stuff into a queue, and insert from the queue in batches.
Earlier quoted context omitted.
Ah cool, thanks for the link! For others, the short-ish answer is that doing hundreds of SQL queries in response to a request (loading nested timeline elements in their case) in SQLite is fine because of the lack of networking/IPC overhead. The nature of N+1 queries is unchanged.
The other half of it that the sqlite page doesn't mention is that sqlite lacks query engine optimizations which would make one large query faster than many smaller queries. If you had a hypothetical in-process version of postgres which didn't have any IPC or networking overhead you'd still see benefits from performing fewer larger queries when using it because the query planner adds some overhead to small queries but…
General SQLite question for the group… I’m making a FOSS analytics system, and ease-of-installation is important. I want to send event data to a separate SQLite database, to keep analytics data separate from the main app’s data. I’m concerned about scaling, since even a modestly busy website could have 1000+ events per second. My thought is to store events in memory on the server and then make one batched write every…
[0] https://clickhouse.com/blog/asynchronous-data-inserts-in-cli...