I wonder if SQLite will ever be modified to allow concurrent writes. Right now it locks the whole DB on write [1]. But since SQLite usage is rapidly expanding from its embedded db roots, one has to wonder if the devs have considered creating some sort of SQLite daemon, or if the underlying architecture would have to be changed so drastically it wouldn't be worth it.
Distributed SQLite: Paradigm shift or hype?
31–40 of 165 posts
Re: Distributed SQLite: Paradigm shift or hype?
#32Earlier quoted context omitted.
[flagged]
Out of curiosity, did I stumble on an actual feud of sorts, or since they're a prolific commenter did they just not recognize you? Do you have links to the first four?
I have no idea what's going on here, but I got the same error and really did fix it by updating Chrome and restarting.
Re: Distributed SQLite: Paradigm shift or hype?
#33WAL2 + BEGIN CONCURRENT ...to pretty much eliminate write contention. I know, it sucks these aren't built in yet. Also if we got some sort of router + map reduce helper (Vitess, Citus -like) it'd make massively distributed SQLite a lot more viable. Setups that don't hammer a single master would make all the difference. Postgres' main disadvantage at scale is all the additional machinery required (backups, failover, p…
this does not eliminate write contention, it just moves a bunch of the problem into the application as now you have to resolve conflicts yourself
> just moves a bunch of the problem into the application
Would you mind elaborating
Re: Distributed SQLite: Paradigm shift or hype?
#34Something can be both hype and useful. Paradigm shift, no, because there's no fundamental architectural difference from any other replicated sql database.
Sqlite only really works as a throwaway database. For that it's pretty great. Although a lot of the current use cases for sqlite could have just been a .ini file.
Re: Distributed SQLite: Paradigm shift or hype?
#35Something can be both hype and useful. Paradigm shift, no, because there's no fundamental architectural difference from any other replicated sql database. Sqlite only really works as a throwaway database. For that it's pretty great. Although a lot of the current use cases for sqlite could have just been a .ini file.
[deleted]
Re: Distributed SQLite: Paradigm shift or hype?
#36this page just says 'Forbidden'
Yeah, says Forbidden in firefox or curl but loads in chromium - welcome to the open web of browser interoperability etc etc where servers now just decide your user agent or tls handshake or whatever is fishy.
also doesn't work on desktop safari for me
Re: Distributed SQLite: Paradigm shift or hype?
#37Earlier quoted context omitted.
Yup, exactly. Phones change wifi networks, routers drop packets, load balancers get overloaded. Hard to fully eliminate tail latencies.
The key here is to make a single API call to the backend which then runs 100+ SQL queries at once and combines the results into a single JSON response - that way you're only paying the network cost once. See https://www.sqlite.org/np1queryprob.html I've implemented GraphQL on top of SQLite and found it to be an amazingly good match, because the biggest weakness of GraphQL is that it makes it easy to accidentally trig…
that doesn't eliminate need for loading states at all, and is already solved by things like react server components or remix (see waterfall removal graphic https://remix.run/)
i think this discussion is confusing the use of sqlite in local-first apps, where there's no loading states because the database is in the browser. you can use sqlite on your server, but you still need a "loading state".
even with postgres, if your data and server are colocated, the time between the 2 is already almost 0
now maybe the argument is your servers are deployed globally each with an sqlite db. that's not all that different from global postgres read replicas
Re: Distributed SQLite: Paradigm shift or hype?
#38Earlier quoted context omitted.
The key here is to make a single API call to the backend which then runs 100+ SQL queries at once and combines the results into a single JSON response - that way you're only paying the network cost once. See https://www.sqlite.org/np1queryprob.html I've implemented GraphQL on top of SQLite and found it to be an amazingly good match, because the biggest weakness of GraphQL is that it makes it easy to accidentally trig…
that doesn't eliminate need for loading states at all, and is already solved by things like react server components or remix (see waterfall removal graphic https://remix.run/ ) i think this discussion is confusing the use of sqlite in local-first apps, where there's no loading states because the database is in the browser. you can use sqlite on your server, but you still need a "loading state". even with postgres, if…
If the data and the server are colocated on the same machine, the database network overhead is almost zero. But that's not necessarily true otherwise. For an HTTP request that incurs a single query, you can round it to zero, but you have to be careful not to write request handlers that incur lots of queries if you're trying to stay below the 100ms threshold.
Re: Distributed SQLite: Paradigm shift or hype?
#39This is kind of missing the point. I can only speak for the company that sponsors LiteFS (I don't think any edge computing companies put 'otoolep up to doing rqlite, for instance). We love Postgres. Most of our users use Postgres. And Postgres works fine in edge/backhaul configurations; companies were doing geographically distributed Postgres read replicas long, long before we came around. That's where we got the ide…
Man! I can’t agree more. It does seem foreign right now, but I truly do believe edge computing will become another “CDN” type abstraction, but it will take time for folks to catch up.
Can’t wait for the day that projects like litefs are just a default that nobody knows about, lol.
Go from “technology nobody knows about” to “technology nobody knows about, but runs the world.”
Re: Distributed SQLite: Paradigm shift or hype?
#40Also I think sqlite would not be a good fit if any sort of slow workers / background jobs are required.