Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Electric (Postgres sync engine) beta release

electric-sql.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Electric (Postgres sync engine) beta release

#51

Post by mrlase »

aidos wrote: post #38
williamstein wrote: post #20

This is really amazing as is their standalone pglite project, which is a WASM port of Postgres. One surprise for me though is that Electric is a read only replica. In particular, “Electric does read-path sync. It syncs data out-of Postgres, into local apps and services. Electric does not do write-path sync. It doesn't provide (or prescribe) a built-in solution for getting data back into Postgres from local apps and s…

We hit on exactly the same pattern for our product and it works really well. We use Hasura as the read engine. That updates a graph of mobx objects that drive the ui. We apply updates directly to those objects so the ui updates immediately. The mutations are posted back to a Python api that applies them to the db. I’ve looked at Electric because we’ve had to recreate some of what they do to interface with Hasura. At…

Would love to read more about how you have mobx structured here. I have a similar graph of mobx objects for something I’m building but haven’t come up with an ergonomic sync story yet.

Re: Electric (Postgres sync engine) beta release

#52

Post by kylemathews »

Ineentho wrote: post #47
thruflo wrote: post #23

Earlier quoted context omitted.

Hey, one of the things here is to define shapes that are shared. If you imagine syncing a shape that is that user’s data then it may be unique. But if you sync, say, one shape per project that that user has access to and a small shape of unique user data then you get shared cache between users who have access to each project. It’s worth noting that Electric is still efficient on read even if you miss the CDN cache. T…

I'm curious on how you'd configure this. Is it common (and safe) to let a cdn cache private data for authenticated users? Say Jira used electric, would you be able to put all tickets for a project behind a cdn cache key? You'd need a cdn that is able to run auth logic such as verifying a jwt to ensure you don't leak data to unauthorized users, right?

Another option too for scaling reads is just putting an nginx in your cluster.

Electric itself is quite scalable at reads too so for a SaaS use-case, you might not need any http proxy help.

Re: Electric (Postgres sync engine) beta release

#54

Post by matlin »

vinner_roy wrote: post #21

Eagerly awaiting the React Native compatible version. https://zerosync.dev/ is the main alternative I'm considering.

You should check out Triplit as well, we have React Native support[0] and the best Typescript integration by far

0. https://www.triplit.dev/docs/frameworks/react-native

Re: Electric (Postgres sync engine) beta release

#55

Post by zknill »

The ElectricSQL concept of 'shapes' is interesting. Right now, a shape is just a single table with some column filters and a where clause. The shape is defined on the fly in the query string. Electric does single table sync of those 'shapes' a lot like Supabase realtime.

I work as part of the team building Ably LiveSync, a competitor in this space. Our postgres connector[0] works in the opposite way by taking rows from an 'outbox' table and fanning them out to subscribed clients over websockets in realtime.

Here's the thing; for any meaningfully complicated data in a relational database the data is likely to be normalised across tables with relations, and there are going to be joins. So I'm really curious how people are making single-table-sync work. (Maybe this is where Electric imagine the future of shapes, solving for joins).

In LiveSync we do it the other way around, instead of having a live-query style subscription to a table (like ElectricSQL), we listen for someone writing a row to the 'outbox' table and that row is automatically sent to subscribers. This means that you're writing your denormlised data directly to the outbox and it's being sent on to clients, rather than writing your normalised data to tables but being limited by single table sync. Or worse, your clients having to subscribe to multiple sync streams and trying and stitch the data back together. We opted to have the write-side insert the denormalised data, rather than having the read side have to stitch normalised data back together.

Electric will at some point try and solve multi table sync, which isn't easy given how the Postgres replication protocol works. It's also not easy to imagine how shapes (which are defined on the fly right now in the query string) will adapt to multiple tables. There's going to be a tradeoff between a complex query string trying to stitch the normalised tables back together, or a 'shape' becoming an entity in Electric which would define how to stitch the normalised tables back together (which you would have to CRUD manage, and update in Electric every time your schema changed).

[0]: https://ably.com/docs/livesync/postgres

Re: Electric (Postgres sync engine) beta release

#56

Post by gejose »

Hey this is great to see, congrats on the launch!

I've been researching 'local first' solutions like electric recently and tried out powersync, triplit and instant for now. All three of these solve for both reading and writing to databases, with offline support.

Wondering if you have plans to support writes too.

Re: Electric (Postgres sync engine) beta release

#57

Post by matharmin »

zknill wrote: post #55

The ElectricSQL concept of 'shapes' is interesting. Right now, a shape is just a single table with some column filters and a where clause. The shape is defined on the fly in the query string. Electric does single table sync of those 'shapes' a lot like Supabase realtime. I work as part of the team building Ably LiveSync, a competitor in this space. Our postgres connector[0] works in the opposite way by taking rows fr…

In my experience, the "outbox" approach means a lot more manual work for developers. It requires developers to create a message for every type of change they want to sync to the client, and then also interpret that on the client. ElectricSQL's Shapes does a lot more work to keep the shape in sync between the client and the server, reducing the need for the developer to do that work.

You're right that "single-table sync" does have its limitations. At PowerSync we effectively support one level of "joins", and even then it's often not enough for more complex schemas. An older version of ElectricSQL did also actually have multi-table shape sync support, but I believe doing that at scale proved to be difficult.

One solution to this is often denormalizing data - either adding more denormalized columns in the existing table, or creating new tables dedicated to sync data. Conceptually, keeping these tables up to date is not that different from writing updates to an outbox table.

I'm also interested in seeing what Zero comes up with in the space. They seem to have solved doing multi-table query sync, but it remains to be seen how well that works in practice.

Re: Electric (Postgres sync engine) beta release

#58

Post by paulddraper »

3PS wrote: post #14
klabb3 wrote: post #10

I have followed a lot of these projects because I have clients with persistent state and synced state too, as well as realtime needs. Anyway, this passes the smell test imo. It’s solid engineering and built from first principles, reusing the right pieces. So congrats! My concern with DB startups is always the business model. There’s a massive tension between open source and a sustainable business which is much more p…

It doesn't always work, but I like the SQLite model: the core offering is free and open source, but enterprises can pay for things like * Professional support, including on-prem hosting when applicable * Additional features that enterprises care about (encrypted databases, SSO) * Compliance documentation/certifications

FWIW SQLite has 3 developers, and I don't think it's even full-time for all of them.

This can work because SQLite is deliberately a very small yet very high impact project.

Very few projects (unfortunately) can boast that.

Re: Electric (Postgres sync engine) beta release

#59

Post by aidos »

thom wrote: post #39
aidos wrote: post #38

Earlier quoted context omitted.

We hit on exactly the same pattern for our product and it works really well. We use Hasura as the read engine. That updates a graph of mobx objects that drive the ui. We apply updates directly to those objects so the ui updates immediately. The mutations are posted back to a Python api that applies them to the db. I’ve looked at Electric because we’ve had to recreate some of what they do to interface with Hasura. At…

Have you looked into something like Materialize? That’d give you realtime updates over even materialised views, works to back Hasura etc.

Hadn’t seen that one. Thanks for the heads up. It’s the sort of thing that I could see being useful in a bunch of places. We have a number of hand rolled things to deal with similar issues.

Might not be a perfect fit here because some of the views are session context dependent (but could maybe still be workable).

Re: Electric (Postgres sync engine) beta release

#60

Post by kylemathews »

gejose wrote: post #56

Hey this is great to see, congrats on the launch! I've been researching 'local first' solutions like electric recently and tried out powersync, triplit and instant for now. All three of these solve for both reading and writing to databases, with offline support. Wondering if you have plans to support writes too.

Check out our guide on writes — https://electric-sql.com/docs/guides/writes

There's a variety of valid patterns for writes & we don't want to be prescriptive about how you do them. We aim instead to help you easily use Electric with your preferred/required write pattern.

It's possible we'll write our own write library but it's equally likely we'll continue to find it better to partner with other projects building excellent libraries (like we are with https://livestore.dev/).

Post reply on HN