Very cool. I haven’t quite worked out how, but I wonder if you could use this for writing really fast integration tests somehow for CI workflows, using some subset of real (or I guess more like curated) data. Also love that the sync layer is Elixir, what a great language.
Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
121–130 of 173 posts
Re: Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
#122Earlier quoted context omitted.
Hey, thanks :) We have a Discord if you fancy chatting at more length! https://discord.electric-sql.com Re: CRDTs and intuition, yup, there's trade offs. We are working hard to deliver a solid model to code against. So we have finality of local writes and standard relational integrity guarantees. You can read a bit more about some of techniques here https://electric-sql.com/blog/2022/05/03/introducing-rich-cr... Re:…
Wouldn't making a backend service be better all around? All CRDT data could be stored elsewhere. The service could be easily updated and changed, and without changes the database. Any database could be used. Could have LDAP integration and other services. I'm sure I don't understand CRDTs. Can they be used with database changes that are not part of the CRDT history? That is, can they be added to an existing database…
A CRDT is a discrete entity/object that has to follow a set of rules for reads, writes, and storage -- including, but not limited to, rules about how to maintain history. You can certainly mix-and-match CRDTs with other entities in the same database, but you can't like "fold in" arbitrary changes to a CRDT without following the rules.
Re: Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
#123Earlier quoted context omitted.
I'm not asking for the problem of key derivation to be solved by ElectricSQL per se, just wondering if it can handle E2E encrypted data in any way. That said, I disagree with how difficult it is: there are a lot of approaches that are not too ridiculous. For example, a simple approach to E2EE is to use a PAKE like SRP to do password authentication, then since the password is kept secret, you can use a normal KDF to d…
I don’t disagree with your technical assessment per se but I would would not label any of this “pretty simple”, except for special purpose apps with technically competent users. Academically it’s in a decent state, but tech- and UX wise I’d say it’s immature. It virtually penetrates all layers of a normal stack (including layer 8). For apps it’s at least possible but the web is very tricky as a platform for anything…
I do agree that the UX of a lot of E2EE software is pretty bad, and if you wanted to make e.g. a chat application, then yeah, it's not necessarily "simple". In practice it can become quite a mess. And honestly, I think a lot of apps have just done a poorer job than what would've been possible without much hassle because it's relatively new territory for a lot of developers.
On the other hand though, we're talking in the context of local-first apps, and the set of constraints imposed by local-first apps already does limit you substantially. If anything, local-first is more complicated in a lot of regards; in my mind, the worst part about E2EE is trying to keep track of all of the different keys you wind up needing to wrangle, whereas with local-first, usually, you're all-but guaranteed to run into CRDTs first-thing, which impose a whole bunch of limitations on the structure of data and the operations you can perform on that data. It is possible to share things in a local-first app, but there are limitations to how much a CRDT-based app can scale for a single document.
And that's why arguably, it's one of the perfect places to add encryption. If you're already talking about apps that deal with small-scale documents and that necessarily do most of the work on the client, you may as well encrypt it. All of the clients are "equal" as far as the data structure goes and the server(s) mainly exist to sync to; it's perfect.
When thinking about the E2EE UX of local-first apps, I think of password managers and not chat applications. I think it's closer to the former than the latter in terms of constraints.
Re: Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
#124This looks incredible. I can certainly see this gaining traction with increased support for other DBs and clients. Great work from these folks!
Re: Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
#125Does it support partial sync ?
Re: Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
#126I see OPFS being mentioned a few times by team members. I could not find any estimates on the website, but could any of the folks in the know share any details? I work on an Electron-based text editor which uses the filesystem to manage text files. Something like this would really be useful to me if it supported plain text files.
Re: Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
#127I'm curious, CRDTs are good for having data converge in the same state but not necessarily ensuring that the state it converges to is "correct". How does this solve that?
You can't guarantee "correctness" (if you define that as zero information loss and/or external causal consistency) in any multi-leader system partly because of unsynchronized clocks and partly because, well, you have to deal with the conflict somehow. You can think of CRDT as a tool to minimize relevant information loss (ideally to zero) when two concurrent writes are converged.
I get that which is why I was curious how ElectricSQL got this to work because SQL databases generally do have a "correct" state.
Re: Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
#128demo request: TodoMVC. No cheating!
Re: Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
#129Can you talk about the dataset sizes you support? Your example linear app has 112 rows, could you support 1000? 10K? 100k? 1M?
OP here, I work for Electric, Electric is designed to support partial sync, and so you don’t have to sync your whole dataset. (Note that this is feature is under development and not yet public) There are limitations on how much data a browser will store for an individual site, so the number of rows you can sync will depend on the shape of your dataset. Finally there are also some performance considerations with WASM…
You’re saying you have partial sync already working for browser sized datasets say 5k rows which are an arbitrary subset of server records which are much larger? What limitations are there on the query criteria for client-synced records, can the subset be expressed as an arbitrary join? How do you implement local first on arbitrary join?
Re: Show HN: ElectricSQL, Postgres to SQLite active-active sync for local-first apps
#130Been looking for something like this. I came across https://www.powersync.co the other day, which looks like similar goals.
Would love to see something like this natively for iOS.