Live data from Hacker News

Distributed SQLite: Paradigm shift or hype?

kerkour.com

151–160 of 165 posts

Re: Distributed SQLite: Paradigm shift or hype?

#151

Earlier quoted context omitted.

Unless your database is in the browser, you are always going to be at mercy of network latencies talking to the backend.

You can with sqlite in webassembly

In theory, but there are still a lot of sharp edges. e.g. I wanted to use sqlite in-browser with an ORM, but few ORMs support such a setup.

Re: Distributed SQLite: Paradigm shift or hype?

#152

Earlier quoted context omitted.

You've been able to build these apps for years now, hell PouchDB was released in 2012. I don't think anything significant is going to push for more of these apps--the DX isn't really that great compared to remote write--there are more abstractions, less caching, way more corner cases, and I call BS on your conflict free utopia.

I'm using Pouch/Couch on a web app, and the sync is really good DX - I got all the functionality I wanted, including easy undo, in a day or so. So yeah, what more DX would actually make people adopt local-first? I don't think it's a dev problem, but a business one. No consumer is demanding local-first, and no company wants to give up that profitable data. We're doing it because we're purposely not interested in users…

Local First (afaiu it) doesn’t have to focus on privacy, though it definitely enables it as a product use case. Outside the privacy focus, it doesn’t remove access to the profitable data (only delays it) while enabling user access patterns that aren’t reliant on 24/7 availability of remote data stores. I think there’s a clear business/product benefit here in that consumer app usage and performance aren’t tied to the users current network quality, and service availability requirements can be less stringent which directly translates to $$ (which can be part of everyone’s value). It probably introduces unique tradeoffs though, and I expect you have a good idea of what those are given you use the pattern. I imagine a data heavy local app syncing with a remote backend unpredictably is its own headache.

WRT Pouch/Couch, and this take is probably annoying to a user of it like you, but it’s an ecosystem that you need to have a reason to get into - IOW it ain’t your grandmas SQL :)

Re: Distributed SQLite: Paradigm shift or hype?

#154
post #93

If you're willing to accept eventual consistency (a big ask, but acceptable in some scenarios) then there are options like marmot [1] that replicate cdc over nats. [1]: https://github.com/maxpert/marmot

The major downside to marmot right now is schema changes don't propagate. Otherwise, keeping a very close eye on it.

It is a pain, but if your use case allows for scheduled downtime, it's not terrible.

I'm looking at Marmot for high availability. Not necessarily horizontal scaling, but instead having a backup server or two that have a constant up-to-date copy of the live db but also can take over if the main server dies, and the main server can then sync the data back when it comes online.

Re: Distributed SQLite: Paradigm shift or hype?

#155
post #33
post #30

Earlier quoted context omitted.

this does not eliminate write contention, it just moves a bunch of the problem into the application as now you have to resolve conflicts yourself

> just moves a bunch of the problem into the application Would you mind elaborating

start here: https://www.sqlite.org/cgi/src/doc/begin-concurrent/doc/begi...

Re: Distributed SQLite: Paradigm shift or hype?

#156

Earlier quoted context omitted.

It is a fairly low level abstraction, but one that does not require a verbose api. There is nothing error prone or hackish about what you have written, it will work for all inputs, it is just low level. You are just used to having other people write this code for you and give you a library. With newer versions of SQLite you could also write CAST(strftime(“%Y”, game_date)) as INTEGER Which is somewhat higher level and…

That's much better, thanks. In case I ever need to do years 10000 :-) Still, having that all over a query looks ugly. SQL is can be unreadable enough as it is without all the joins/table renaming. I just want something more readable like EXTRACT(year from date), like you can in Postgres et al. Would also be nice if there was a native timestamp like there is in, pretty much every other database. I'm sensitive to "feat…

I agree it's less obviously correct, but I bet you could add the extension to sqlite if you feel strongly about it. As an aside '%y' is documented to only work in sqlite for years >= 0000 and for instance `select date(-50000000000, "unixepoch");` returns `0385-07-25`

Interestingly %Y doesn't seem to handle negative dates either if you need to handle BC, so I guess that is one downside for both. This is one reason I sometimes prefer to use low level code even when it is less obviously correct with a cursory glance, because abstractions may not mean what you think they mean, or even worse, may be lying to you. At least with low level code I can reason about how it would behave under certain edge cases I might care about.

Re: Distributed SQLite: Paradigm shift or hype?

#157

Earlier quoted context omitted.

>most SQL will carry over between SQLite and Postgres and MySQL, especially if you add ORMs in the mix I think this goes underappreciated, or rather the opposite is overstated. Sure there are some edge cases that don't work the same, but most apps won't hit those. My _biggest_ gripe with SQLite so far is the lack of column reordering like other DBs. And my simplistic understanding is that the others do it exactly the…

> Sure there are some edge cases that don't work the same, but most apps won't hit those. That really depends on your modelling style. If you like things like types, SQL-side processing (eg using functions), or covering indexes, then you’ll hit issues every five minutes in sqlite. SQLite really wants the logic (including consistency logic) in the application, just compare the list of aggregate functions in postgres v…

I use SQLite in my personal projects, not professionally. I was wondering if you could elaborate on what you mean by 'consistency logic' in the context of SQLite.

Re: Distributed SQLite: Paradigm shift or hype?

#158
post #70

Earlier quoted context omitted.

You can run postgres on the same host as the web server too. Isn't that going to get you most of that same benefit in speed?

Postgres is great, but managing a fleet of them (one in each web server) and ensuring they are all working fine would bring a lot of operational complexity. SQLite, on the other hand, skips all of that with its simple in-process model.

I just start postgres and it's solid. I don't understand this statement. With Dokku it's trivial.

Re: Distributed SQLite: Paradigm shift or hype?

#159
The problem with Postgres is that out of the box, it is not much different from SQLite in terms of availability and scaling. You need to add a Pgbouncer before it can handle multiple connections, and you need a really complicated dance to make it highly available, with read replicas and hot standbys. This is why I prefer MongoDB for that matter.

Re: Distributed SQLite: Paradigm shift or hype?

#160

Earlier quoted context omitted.

I'm using Pouch/Couch on a web app, and the sync is really good DX - I got all the functionality I wanted, including easy undo, in a day or so. So yeah, what more DX would actually make people adopt local-first? I don't think it's a dev problem, but a business one. No consumer is demanding local-first, and no company wants to give up that profitable data. We're doing it because we're purposely not interested in users…

Local First (afaiu it) doesn’t have to focus on privacy, though it definitely enables it as a product use case. Outside the privacy focus, it doesn’t remove access to the profitable data (only delays it) while enabling user access patterns that aren’t reliant on 24/7 availability of remote data stores. I think there’s a clear business/product benefit here in that consumer app usage and performance aren’t tied to the…

You're right, as I understand it myself, it's more about overcoming network burps, a major point of user friction. I guess local-first is not the exact term for what I'm thinking, which is that an app should not need the Internet to work, but it can add conveniences, foremost being able to get to your setup from another device. That's how some of my most used tools like to-do list and notes work. In this scenario, the server could be dumbly storing the data without understanding it, and the user could point to another data store.

I'm generally a Postgres user not because I love SQL but because I hate drama. That said, my Pouch/Couch use case is fairly simple and probably well aligned with the intent so far and so the paradigm clicks for me.

Post reply on HN