Live data from Hacker News

Show HN: Bi-directional sync between Postgres and SQLite

powersync.com

21–30 of 107 posts

Re: Show HN: Bi-directional sync between Postgres and SQLite

#21
post #16

I couldn’t tell whether I could integrate my SQLite app with my own Postgres database? E.g. is it free to self host, and the paid services are a convenience? Or the cloud hostage is mandatory?

You can use any Postgres database as long as it supports enabling logical replication and PowerSync can connect to it. Details here: https://docs.powersync.com/usage/installation/database-setup The PowerSync Service is currently a hosted cloud service. A self-hostable version is coming soon.

Ah thanks. Out of curiosity, will the self-hostable version be a paid product, or free to use? (No judgement, just want to understand the business model).

Re: Show HN: Bi-directional sync between Postgres and SQLite

#23

Thank you - it’s great to have an alternative to electric-sql. I actually need a local first, disconnected framework. What many people don’t realize is that even in the US, there are many parts without any network connectivity like our parks and preserves. Rangers still need to make queries in those areas.

You can try Watermelon DB. It is local first disconnected framework. And it has a sync framework as well, but you have to create your own backend (using any DB) for syncing. Stable product and works great

https://watermelondb.dev/docs/Sync/Intro

Re: Show HN: Bi-directional sync between Postgres and SQLite

#24
post #16

Earlier quoted context omitted.

You can use any Postgres database as long as it supports enabling logical replication and PowerSync can connect to it. Details here: https://docs.powersync.com/usage/installation/database-setup The PowerSync Service is currently a hosted cloud service. A self-hostable version is coming soon.

Ah thanks. Out of curiosity, will the self-hostable version be a paid product, or free to use? (No judgement, just want to understand the business model).

The initial self-hostable version will be available as part of the paid service, but there is also a free open-source self-hostable version planned for next year:

The PowerSync Service will move to an open-core model in 2024: An open-source version will be available (self-hosted) as well as a non-open-source version with premium features (available as both self-hosted and hosted cloud service)

The client SDKs are currently open-source (Apache 2.0 license)

Re: Show HN: Bi-directional sync between Postgres and SQLite

#25
post #5
post #4

Earlier quoted context omitted.

SQLite has a number of functions to work with time but the underlying storage format is just going to be--depending on what precision/range of timestamp you want--a float or an integer... to the extent, of course, that SQLite has types at all (as it frankly doesn't, thereby making this question kind of moot).

SQLite has five types, documented here: https://www.sqlite.org/datatype3.html null, integer, real, text, blob It has a historically cavalier attitude to enforcing them (which I believe it inherited from TCL) but that changed in November 2021 with the release of strict table mode in version 3.37.0: https://www.sqlite.org/stricttables.html

JSON is a sixth type in practice due to the way `text` acts differently depending on whether it is fed directly from one JSON function to another.

See https://sqlite.org/json1.html#value_arguments

Re: Show HN: Bi-directional sync between Postgres and SQLite

#26

Great concept! What if the user is offline, makes updates to the local/embedded DB, but when it gets online there's an issue in syncing that prevents updating the postgress with the SQLite changes? Is SQLite reverted to postgress and thus the changes lost?

The client does not update its state to the authoritative state of the server as long as there's pending writes present in the client's upload queue. If there is an error with uploading a write to the server (either a network error or the server returning an HTTP error response), the write will remain in the upload queue and the upload will be retried.

If the server can automatically resolve an error or conflict, it would return a 2xx response and the client's state will be updated to match the server's authoritative state.

If the conflict/error cannot automatically be resolved, there are a few options:

- The server can return a 2xx response but record information about the conflict that allows a user to resolve it (e.g. prompt a user to manually resolve the conflict — either the end-user or administrator for example)

- The server can save details in a dead letter queue

More details are documented here:

https://docs.powersync.com/architecture/consistency

https://docs.powersync.com/usage/lifecycle-maintenance/handl...

Re: Show HN: Bi-directional sync between Postgres and SQLite

#27
Since the company seems to be answering questions I’ll give a few that come up for me. Congrats on the launch btw.

In the given design, am I understanding correctly that this means that a local commit could be seen as “committed” by the user but then later the server rejects it because of conflicts right? I guess it’s all application defined in that you could build your application in such a way as to show enqueued but uncommitted changes differently from the checkpoint, but it does mean that you could lose data if the application author doesn’t handle that well in their local code or the application server yeah? Not a critique because I think that’s true even for anything handrolled, but just making sure I understand the model.

Also, I’m a little unclear how it’s tied to postgres and what the powersync service is doing. The docs say that you fetch a JWT from your application to talk to the service but then it says that all writes are mediated by your own application code. So presumably the powersync service channel is for synchronization of Postgres -> local SQLite. Is that right? And the replication logic in powersync - is that essentially accomplishing horizontal sharding of the database for reads? Also, for the replication piece is the SQLite bit actually important or could you actually support arbitrary backends and SQLite is just convenient? Eg could you support browser LocalStorage instead of WASM SQLite or is there some piece of functionality of SQLite you’re relying on?

Finally, do you have any support for lazy local hydration / eviction? Eg if I have a Google docs like application, is it synchronizing my entire account at all times or does it support pulling a document and evicting LRU documents when some local storage limit is exceeded?

Re: Show HN: Bi-directional sync between Postgres and SQLite

#28

Heads up: The submission link https://powersync.com/ results in ERR_ADDRESS_UNREACHABLE You linked the proper landing page here: https://www.powersync.com/ Your web infrastructure doesn't redirect root visitors to www (or pass those requests on to your web server)

That's strange, we are unable to reproduce this on our end — both URLs load correctly for our team. I'll shoot an email to the address in your bio to see if there's any details you could perhaps share that could help us debug. Thanks for letting us know :)

Re: Show HN: Bi-directional sync between Postgres and SQLite

#29
post #10

Earlier quoted context omitted.

The goal is basically to help you build offline-first apps. In the offline-first paradigm, app code works directly with a client-side embedded database, which automatically syncs with a backend database in the background. This is in contrast to cloud-first apps which primarily use a cloud datastore via APIs. Working with a local database (e.g. SQLite) means that apps feel instant to use because of low latency, and re…

Oh that's very interesting. I guess it's easier to deal with "only" sqlite than to have a store, keep track of mutations, and then upload/download the mutations when the device is reconnected.

For sure, working only with a local SQLite database does simplify state management in many cases. For the most part you don't need an additional state management system/library.

Meta had an interesting article a few years back about how they rewrote the Messenger app to use SQLite: https://engineering.fb.com/2020/03/02/data-infrastructure/me...

Re: Show HN: Bi-directional sync between Postgres and SQLite

#30
I am looking for something like this. We develop IoT devices, thousand of linux computer running behind customer routers and have a central server with web admin, all in python.

I wish i could have "syncronized objects", something were a change in a device would be reflected back to the server, web interface even better and in the other direction too, a way to see the whole system as one thing, not a hodge podge of api, calls, update and so on.

Is there an elegant way to do this?

Post reply on HN