Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beyond the SQLite single-writer limitation with concurrent writes

turso.tech

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Beyond the SQLite single-writer limitation with concurrent writes

#41

Post by zeroq »

crazygringo wrote: post #22

While I understand that you could do this, I genuinely don't understand why. SQLite and Postgres/MySQL/etc. occupy different niches. If you need massive concurrent writes, surely that's what Postgres/MySQL/etc. is for? Their engines are built around that from the ground up. SQLite is built around a file that stores data for a single application, as opposed to being client-server with many clients. I've used it a ton…

I'll give you an example of what I'm doing with SQLite.

I have an app that organizes my movie collection. I use IMDB public datasets as data source. Because versioning is hard, data changes over time (remember who directed Matrix?) it's just easier to rebuild the database every time.

A naive import of datasets is ~131M inserts. It takes 10 minutes (of which 100M takes exactly 6 minutes). Another 10 minutes to further clean up the data, set up indexes, optimize and vacuum.

It's fine for the use case, but still it would be nicer to be able to achieve that in half the time.

The app has no dependencies, database is a single file I can store on S3 and with a little bit of magic use it as a real database - meaning I can run a complex queries over http on that whole database and exchange 50kb.

The client is a standalone HTML file with some JS. No need of server for client, no need for hosting for server except of that S3 bucket.

Sure, it's definitely a niche application, but, at least for me, it makes perfect sense. Start your app with zero deps and add only what you really need. There's too much projects that start with google-esque architecture, with plenty of microservices, only to reach MVP with technical debt the size of US treasury.

Re: Beyond the SQLite single-writer limitation with concurrent writes

#42

Post by glommer »

fragmede wrote: post #33
glommer wrote: post #30

Earlier quoted context omitted.

You just have to pass the experimental flag. I thought this was explained in the blog ?

I searched https://turso.tech/blog/beyond-the-single-writer-limitation-... with ctrl-f for "download" and "--exp" and got no results for either. There was a link to a live demo webpage at the very end. Am I looking at the right blog? That's the one linked at the top of the page.

The flag is --experimental-mvcc, it is in the help for the shell. You are right that the blog failed to mention. I will get this fixed soon!! Thanks for noticing it.

Re: Beyond the SQLite single-writer limitation with concurrent writes

#43

Post by CyberDildonics »

zeroq wrote: post #41
crazygringo wrote: post #22

While I understand that you could do this, I genuinely don't understand why. SQLite and Postgres/MySQL/etc. occupy different niches. If you need massive concurrent writes, surely that's what Postgres/MySQL/etc. is for? Their engines are built around that from the ground up. SQLite is built around a file that stores data for a single application, as opposed to being client-server with many clients. I've used it a ton…

I'll give you an example of what I'm doing with SQLite. I have an app that organizes my movie collection. I use IMDB public datasets as data source. Because versioning is hard, data changes over time (remember who directed Matrix?) it's just easier to rebuild the database every time. A naive import of datasets is ~131M inserts. It takes 10 minutes (of which 100M takes exactly 6 minutes). Another 10 minutes to further…

I don't understand what you are trying to say here, this doesn't seem to have anything to do with single writer limitations.

Re: Beyond the SQLite single-writer limitation with concurrent writes

#44

Post by fragmede »

glommer wrote: post #42
fragmede wrote: post #33

Earlier quoted context omitted.

I searched https://turso.tech/blog/beyond-the-single-writer-limitation-... with ctrl-f for "download" and "--exp" and got no results for either. There was a link to a live demo webpage at the very end. Am I looking at the right blog? That's the one linked at the top of the page.

The flag is --experimental-mvcc, it is in the help for the shell. You are right that the blog failed to mention. I will get this fixed soon!! Thanks for noticing it.

Ah there it is! Thanks, I'll play around with it tonight.

Re: Beyond the SQLite single-writer limitation with concurrent writes

#45

Post by glommer »

frumplestlatz wrote: post #37
glommer wrote: post #32

Earlier quoted context omitted.

I understood your criticism. I disagree with it, and I think we are not misrepresenting anything, as we make it very clear that we are not SQLite, we are just a reimplementation that goes beyond it (evolution). The disagreement about their contribution model of course happened, but the meaning you ascribe to it, perhaps is something you imagined. It boils down to what you understand "criticism" to be. If I see someon…

You’ve reframed this as a discussion about open versus closed development. That wasn’t the point I raised. My concern is about presentation, ethics, respect, and about the co-option of a gift to the commons — particularly how your public messaging gives readers the impression of lineage and endorsement that doesn’t exist. Regardless of your intent, that’s the effect of calling Turso “the next evolution of SQLite.” Yo…

Yes, this is (one of) the point(s) you raised. You said there was a public an "unpleasant fight" (never happened) because of "criticism of SQLite" on my part. I calmly explained that such thing never took place. I have a preference towards Open models but never criticized SQLite (as in stated that they are wrong). Where does the unpleasant fight comes from?

Your claim that we are doing something ethically wrong seems to be informed by your pre-existing opinion of me, that itself derives from the "unpleasant fight" (that never happened).

As for the tagline we use, most people don't get the impression that there is any violation of ethics or respect. This is evidenced by other people's reaction here. You do, and you are within your right. I can't, unfortunately, please everybody.

Some people are more relevant than others, though: in this case, if the authors of SQLite expressed their opinion to me that this crosses a line in their view, I'd change it, without blinking an eye.

I have a tremendous respect for them, and we want our messaging to convey nothing but that!

Re: Beyond the SQLite single-writer limitation with concurrent writes

#46

Post by glommer »

glommer wrote: post #42
fragmede wrote: post #33

Earlier quoted context omitted.

I searched https://turso.tech/blog/beyond-the-single-writer-limitation-... with ctrl-f for "download" and "--exp" and got no results for either. There was a link to a live demo webpage at the very end. Am I looking at the right blog? That's the one linked at the top of the page.

The flag is --experimental-mvcc, it is in the help for the shell. You are right that the blog failed to mention. I will get this fixed soon!! Thanks for noticing it.

There is a lot of work that still needs to be done to make this production-ready, both from a performance and reliability point of view, as we did our best to convey on the blog. We truly appreciate you trying it out! Report any issues, please.

Re: Beyond the SQLite single-writer limitation with concurrent writes

#47

Post by crazygringo »

zeroq wrote: post #41
crazygringo wrote: post #22

While I understand that you could do this, I genuinely don't understand why. SQLite and Postgres/MySQL/etc. occupy different niches. If you need massive concurrent writes, surely that's what Postgres/MySQL/etc. is for? Their engines are built around that from the ground up. SQLite is built around a file that stores data for a single application, as opposed to being client-server with many clients. I've used it a ton…

I'll give you an example of what I'm doing with SQLite. I have an app that organizes my movie collection. I use IMDB public datasets as data source. Because versioning is hard, data changes over time (remember who directed Matrix?) it's just easier to rebuild the database every time. A naive import of datasets is ~131M inserts. It takes 10 minutes (of which 100M takes exactly 6 minutes). Another 10 minutes to further…

That's not concurrent writes. You would see zero speedup.

Your use case makes perfect sense for SQLite. It's a great example of what it's for. It won't benefit. That's my point.

Re: Beyond the SQLite single-writer limitation with concurrent writes

#48

Post by glommer »

simonw wrote: post #10
tracker1 wrote: post #7

Kind of cool to see work on this. I do hope that the final db file result is still binary compatible with SQLite 3 in whatever direction Turso moves towards though... Rust or not. I've been advocating with several projects over recent years to get SQLite3 as an archive/export/interchange format for data. Need to archive 2019 data from the database, dump it into a SQLite db with roughly the same schema... Need to pass…

SQLite against S3 can work with some clever tricks. The neatest version of that I've seen is still this WebAssembly one: https://phiresky.github.io/blog/2021/hosting-sqlite-database... I also got sqlite-s3vfs working from Python a few months ago: https://simonwillison.net/2025/Feb/7/sqlite-s3vfs/ Both of these are very much read-only mechanisms though.

simon, you will be pleased to know that the python package for turso is in good shape! (and if you do find any issues please scream at us)

Re: Beyond the SQLite single-writer limitation with concurrent writes

#49

Post by frumplestlatz »

glommer wrote: post #45
frumplestlatz wrote: post #37

Earlier quoted context omitted.

You’ve reframed this as a discussion about open versus closed development. That wasn’t the point I raised. My concern is about presentation, ethics, respect, and about the co-option of a gift to the commons — particularly how your public messaging gives readers the impression of lineage and endorsement that doesn’t exist. Regardless of your intent, that’s the effect of calling Turso “the next evolution of SQLite.” Yo…

Yes, this is (one of) the point(s) you raised. You said there was a public an "unpleasant fight" (never happened) because of "criticism of SQLite" on my part. I calmly explained that such thing never took place. I have a preference towards Open models but never criticized SQLite (as in stated that they are wrong). Where does the unpleasant fight comes from? Your claim that we are doing something ethically wrong seems…

Your focus is on intent; mine is on the ethical consequences of behavior.

The earlier disagreement with SQLite was one-sided because the other side saw no reason to engage. I expect that dynamic will continue.

Re: Beyond the SQLite single-writer limitation with concurrent writes

#50

Post by crazygringo »

glommer wrote: post #39
Quarrelsome wrote: post #36

Earlier quoted context omitted.

Agreed, Sqlite docs specifically state that its not designed for this, or rather that the solution isn't appropriate, that database as a service is appropriate when you want multiple clients writing. The entire "problem" is a side-effect of using the wrong tool for the job.

Linux was designed to run in home PCs, and we keep running it in supercomputers. It works just fine. Tools evolve.

Tools don't always need to evolve though. I don't want my hammer to evolve into a screwdriver too, or vice-versa.

Having separate tools for separate things makes sense when the things are different enough.

Post reply on HN