Live data from Hacker News

Distributed SQLite: Paradigm shift or hype?

kerkour.com

71–80 of 165 posts

Re: Distributed SQLite: Paradigm shift or hype?

#71

SQLite is neither shift nor hype, its a misunderstanding of underlying architecture challenges. SQLite has pros and cons, like anything else. When it comes to web frameworks and app platforms, to fundamental problem is that many committed fully to server less/edge and ignored the los of persistent storage. You just can't use a local database when using serverless, or any type of distributed compute/rendering for that…

[deleted]

Re: Distributed SQLite: Paradigm shift or hype?

#72

LiteFS author here. I don't disagree with any points in the article but perhaps a reframing could help. I previously wrote a tool called Litestream that would do disaster recovery for a single-node SQLite server and I still think it's a great default option for people starting new projects. Unless you're doing very database-specific things, most SQL will carry over between SQLite and Postgres and MySQL, especially if…

>most SQL will carry over between SQLite and Postgres and MySQL, especially if you add ORMs in the mix I think this goes underappreciated, or rather the opposite is overstated. Sure there are some edge cases that don't work the same, but most apps won't hit those. My _biggest_ gripe with SQLite so far is the lack of column reordering like other DBs. And my simplistic understanding is that the others do it exactly the…

Why would you want to recorder columns? SQLite reads in a whole record at a time to access any column.

Re: Distributed SQLite: Paradigm shift or hype?

#73

We have so many distributed X applications nowadays that all try to solve the same problem, either in the same or different ways. I think we first have to come up with a simple, distributed, open-source storage solution. In the cloud, we have things like AWS S3, which is a very reliable distributed storage, but for self-hosting, we have: Ceph, with which I have much experience, is a very solid and quite bulletproof s…

> While MinIO's setup is complex, SeaweedFS's setup is much simpler.

Are you sure you are not talking in reverse?

I find Minio single binary deployment very easy, and you also complained about SeaweedFS's complexity in the previous paragraph.

Re: Distributed SQLite: Paradigm shift or hype?

#74
The paradigm shift that's going to come with distributed SQLite isn't to the edge, it's going to be to users devices. I believe the DX of building Local-first apps is going to hit the criticality point in the next year or so and its popularity is going to explode.

With dynamic partial replication you can synchronise a subset of your database to a SQLite db on your users device, eliminating the network from the ui interaction loop. Then with the emerging eventually constant syncing systems, many using CRDTs, it's possible to have conflict free eventual consistency. Just read and write to a local database and it will sync with your central server or other clients in the background, both in realtime or after working offline.

I work on one such system at ElectricSQL, but there are many people building variants of this such as Evolu, SQLsync, CR-SQLite, and PowerSync.

That's all not to say SQLite on the edge isn't really damn cool!

Re: Distributed SQLite: Paradigm shift or hype?

#75

The paradigm shift that's going to come with distributed SQLite isn't to the edge, it's going to be to users devices. I believe the DX of building Local-first apps is going to hit the criticality point in the next year or so and its popularity is going to explode. With dynamic partial replication you can synchronise a subset of your database to a SQLite db on your users device, eliminating the network from the ui int…

All of this makes me think we'll just be building meteor apps but with sqlite instead of mongo and minimongo whaha

Re: Distributed SQLite: Paradigm shift or hype?

#76

Earlier quoted context omitted.

>most SQL will carry over between SQLite and Postgres and MySQL, especially if you add ORMs in the mix I think this goes underappreciated, or rather the opposite is overstated. Sure there are some edge cases that don't work the same, but most apps won't hit those. My _biggest_ gripe with SQLite so far is the lack of column reordering like other DBs. And my simplistic understanding is that the others do it exactly the…

Why would you want to recorder columns? SQLite reads in a whole record at a time to access any column.

Not true.

Re: Distributed SQLite: Paradigm shift or hype?

#77

Earlier quoted context omitted.

Unless your using database specific features. One of the biggest advantages for Postgres is how incredible the ecosystem is. It doesn't work for everything, but I have an OEM, multiple kinds of text search (vector, inverted indexes, trigrams), recursive and graph-like queries (though that's admittedly less of an issue if n+1 isn't a problem), row-level acls, locks, etc. It's really nice to have all of that power avai…

Not really a problem if you go from sqlite to postgres. Which sqlite feature is missing from Postgres?

This is saying: "just don't try to solve hard data storage problems". Not all applications are CRUD.

Re: Distributed SQLite: Paradigm shift or hype?

#78

> While SQLite is a really amazing database, most teams will benefit from avoiding it and going the PostgreSQL way instead. > Bazillions of engineering hours have been spent to make Postgres the best backend database and choosing SQLite will inevitably force you to reinvent what Postgres already had for many years, in a fragile and buggy way. Could the same not also be said for MS SQL Server, Oracle, Sybase, MySQL, o…

Without giving evidence, I do agree that Postgres is a better database that serves the general case better. I use both in production for different purposes, but postgresql makes a lot more sense then SQLite as our primary database

Re: Distributed SQLite: Paradigm shift or hype?

#79
Don't get caught up in marriage to a particular tech flavor. Instead, architect and use a generic db adapter pattern that translates a more generic SQL/GraphQL flavor into whatever is necessary and efficient underneath for that particular DBMS or store. Meta does this for many warm storage services. It makes changing underlying DBMSes and scalability architectures much easier to develop and reason about than if every app has to know everything about its backend(s).

Re: Distributed SQLite: Paradigm shift or hype?

#80
post #70
post #8

I think this skips one mega benefit for apps. I’ve been using liteFS in production for a couple months. Your web app is able to resolve db queries instantly. You don’t need loading states if you’re using complex charts and other frontend JS that waits for data. All the data is resolved so fast and you can just return all your data like more traditional apps, and the load times are insane. If you’re multi region you c…

You can run postgres on the same host as the web server too. Isn't that going to get you most of that same benefit in speed?

It is. I do this on plenty of hobby Laravel apps.
Post reply on HN