Even in the “low-performing” zone of the single threaded SQLite and Turso, we are still talking about 50k rows per second and 1000 microseconds, aka 1 ms. It is insanely fast. 1ms is just the round-trip for my Postgres on RDS. It is amazing that SQLite is so awesome. I understand it is not for every use-case, but still awesome
Beyond the SQLite single-writer limitation with concurrent writes
21–30 of 75 posts
Re: Beyond the SQLite single-writer limitation with concurrent writes
#22While I understand that you could do this, I genuinely don't understand why.
SQLite and Postgres/MySQL/etc. occupy different niches. If you need massive concurrent writes, surely that's what Postgres/MySQL/etc. is for? Their engines are built around that from the ground up.
SQLite is built around a file that stores data for a single application, as opposed to being client-server with many clients. I've used it a ton for that, but the idea that your application would have so many threads needing to write concurrently that it would be slowed down by a file lock... just doesn't make sense to me.
What applications need this, and why wouldn't they use Postgres/MySQL/etc. if they need such a high level of concurrent performance? This feels like trying to adapt a small sports car to tow a semi-trailer. It doesn't seem like it's what it's meant for.
Re: Beyond the SQLite single-writer limitation with concurrent writes
#23I’m not in a rush to use a reimplementation of SQLite — particularly from startup bros that had a very public, one-sided, and unpleasant fight with SQLite over their contribution model. D. Richard Hipp is a genuinely fantastic human being, and SQLite is a project developed at literally the level of planetary infrastructure given how broadly and everywhere it appears. Forking his project while using the name to garner…
Author of Turso here. Couple of points
* You are right not to rush. You should keep using SQLite until Turso matures. Some use cases are more tolerant to new tech than others. It will take time for us to reach the level of trust SQLite has for broad use cases, but we are hoping to add value for some use cases right away. Never rush, tech matters!
* I have never met Hipp, but only heard great things about him.
* We never had a fight with SQLite over their contribution model (or about anything for that matter, I never even met Hipp or anybody else from SQLite). We just disagree with it - in the sense that we believe in different things. We don't think what they do is fundamentally wrong. Different projects take different paths.
* We are not using the SQLite name. We compare ourselves to SQLite because we are file and API compatible, and we do aspire to raise the very high bar they have set. It is hard to do this without drawing the comparison, but we are a different project and state it very clearly. I am not a lawyer (and neither you seem to be), but we believe we are doing is okay. If we ever have any valid reason to believe we crossed a line here, we will of course change course.
* We are not "startup bros". We spent 20+ years of our lives building databases and operating systems.
Re: Beyond the SQLite single-writer limitation with concurrent writes
#24Can't use it locally (yet?) but it's definitely an interesting move in the space. For my personal projects lately I've been defaulting to sqlite in dev, and having a database wrapper layer to use something else in prod.
why not? Turso is a fully local database, you can just download the shell and use it as you would use sqlite.
Re: Beyond the SQLite single-writer limitation with concurrent writes
#25Very excellent to see the embedded DB in the browser link at the bottom. https://shell.turso.tech/
I've been thinking Turso in the browser would be some eventual addition, is something the Rust rewrite would eventually unlock. And that would require considerable extra backing to make go. But here it is now!
Re: Beyond the SQLite single-writer limitation with concurrent writes
#26The single-writer limitation in SQLite is per-database, not per-connection. You can shard your SQLite tables into multiple database files and query across all of them from a single connection. I agree that "the single-writer limitation isn't just a theoretical concern", but it's also solvable without forking SQLite. ulimit's the limit! If your goal is resource maximization of a given computer, though, Postgres is lik…
Joins and Transactions are a pretty big part of SQL. I'm no expert, but if my quick search results are right, both are lost in the separate file per table scenario.
They work just fine!
Joins: https://simonwillison.net/2021/Feb/21/cross-database-queries...
Transactions: https://www.sqlite.org/atomiccommit.html#_multi_file_commit
Re: Beyond the SQLite single-writer limitation with concurrent writes
#27Can't use it locally (yet?) but it's definitely an interesting move in the space. For my personal projects lately I've been defaulting to sqlite in dev, and having a database wrapper layer to use something else in prod.
why not? Turso is a fully local database, you can just download the shell and use it as you would use sqlite.
More specifically, I didn't see any commits about multiple simultaneous writers having landed in main and it's not document as a feature in the release notes https://github.com/tursodatabase/turso/commits
neither the prerelease
https://github.com/tursodatabase/turso/releases/tag/v0.3.0-p...
nor their latest release.
https://github.com/tursodatabase/turso/releases/tag/v0.2.2
The core/mvcc directory does have commits though, so maybe you can.
https://github.com/tursodatabase/turso/commits/main/core/mvc...
Re: Beyond the SQLite single-writer limitation with concurrent writes
#28I’m not in a rush to use a reimplementation of SQLite — particularly from startup bros that had a very public, one-sided, and unpleasant fight with SQLite over their contribution model. D. Richard Hipp is a genuinely fantastic human being, and SQLite is a project developed at literally the level of planetary infrastructure given how broadly and everywhere it appears. Forking his project while using the name to garner…
Author of Turso here. Couple of points * You are right not to rush. You should keep using SQLite until Turso matures. Some use cases are more tolerant to new tech than others. It will take time for us to reach the level of trust SQLite has for broad use cases, but we are hoping to add value for some use cases right away. Never rush, tech matters! * I have never met Hipp, but only heard great things about him. * We ne…
I wasn’t describing a legal dispute, and I haven’t made any claim about infringement. “Material misrepresentation” refers to the substance of the public messaging, not to a legal violation.
The issue I raised is that the phrasing and the way the fork has been presented create the impression of continuity and endorsement that doesn’t exist. That’s a reputational and ethical concern, not a legal debate. Calling it “the next evolution of SQLite” is, in practice, absolutely trading on the SQLite name.
There was a very public, one-sided disagreement about SQLite’s contribution model at the time, and you’ve been open about your criticisms of SQLite in the years since. That’s the context for my comment; it isn’t something I’ve imagined.
Re: Beyond the SQLite single-writer limitation with concurrent writes
#29How does the mvcc / row versioning changes that accumulate get shared among multiple processes? Is it another file like the WAL file? Or is it just local memory as part of the client? I guess with everything having to commit or entirely roll back at the end, it doesn't need to be shared.
How does this compare with hctree then? It sounds similar, both using an MVCC model.
Really cool though, love the technical details and would be interested to see further. I feel like there is certainly a space for sqlite+fancier features. Though it would break backwards compatibility, I've always wondered what kind of performance gains could be had by using a different varint implementation. And enforcing or always using "strict" for smarter column layouts.
Also would be really cool to have indexed views.
And columnstore seems like a logical direction as well.
Re: Beyond the SQLite single-writer limitation with concurrent writes
#30Earlier quoted context omitted.
why not? Turso is a fully local database, you can just download the shell and use it as you would use sqlite.
More specifically, I didn't see any commits about multiple simultaneous writers having landed in main and it's not document as a feature in the release notes https://github.com/tursodatabase/turso/commits neither the prerelease https://github.com/tursodatabase/turso/releases/tag/v0.3.0-p... nor their latest release. https://github.com/tursodatabase/turso/releases/tag/v0.2.2 The core/mvcc directory does have commits t…
You just have to pass the experimental flag. I thought this was explained in the blog ?