Live data from Hacker News

Show HN: Bi-directional sync between Postgres and SQLite

powersync.com

71–80 of 107 posts

Re: Show HN: Bi-directional sync between Postgres and SQLite

#71
post #63

what happens if you go out of business?

We've been in business for over 10 years, and the company (JourneyApps) is profitable - we're not going out of business any time soon.

Also see the other comments regarding our current status and plans around open source.

Re: Show HN: Bi-directional sync between Postgres and SQLite

#72

Earlier quoted context omitted.

You can do it in a consistent way, for example always store the timestamp at 00:00 UTC. But that's not obvious just by looking at the values, which is why it can be ambiguous. And when parsing those values, you could do for example (JS) `new Date(ts*1000).getDay()`. Of course that's not correct - you need to use the UTC methods. But it's easy to miss, and may pass all your tests until you switch the timezone. Those a…

Not sure I follow. What's the difference between `new Date(ts).getDay()` when ts is an ISO8601 string or a ms since UNIX epoch value. Pretty sure the result is the same and whether you need to use `getDay` or `getUTCDay` is a display issue that affects both variants. I could see it being more convenient if you're just dumping the raw values without any transformation or you don't know what data type is stored in a co…

when you do that, it will give you different days depending on your timezone. 1701388800 is 2023-12-01 in london, but still 2023-11-30 on new york.

Re: Show HN: Bi-directional sync between Postgres and SQLite

#73

Thank you - it’s great to have an alternative to electric-sql. I actually need a local first, disconnected framework. What many people don’t realize is that even in the US, there are many parts without any network connectivity like our parks and preserves. Rangers still need to make queries in those areas.

You can try Watermelon DB. It is local first disconnected framework. And it has a sync framework as well, but you have to create your own backend (using any DB) for syncing. Stable product and works great https://watermelondb.dev/docs/Sync/Intro

Hi everyone, I worked with watermelonDB and recently switched my entire project to powersync. Watermelon has its limitations for an offline applicationFirst, if you really want to make use of the application completely offline, you will have to build a synchronizer like CRDT by hand and manage the request queues. With powersync all of this is managed by them, and with a simple code I can choose which information from the database I will sync for each user. In my first tests with WatermelonDB, synchronization proved to be unfeasible due to the amount of synchronized data. In short, Powersync has proven to be a wonderful tool that has allowed my company to move forward with offline services.

Re: Show HN: Bi-directional sync between Postgres and SQLite

#74

Earlier quoted context omitted.

You can do it in a consistent way, for example always store the timestamp at 00:00 UTC. But that's not obvious just by looking at the values, which is why it can be ambiguous. And when parsing those values, you could do for example (JS) `new Date(ts*1000).getDay()`. Of course that's not correct - you need to use the UTC methods. But it's easy to miss, and may pass all your tests until you switch the timezone. Those a…

Not sure I follow. What's the difference between `new Date(ts).getDay()` when ts is an ISO8601 string or a ms since UNIX epoch value. Pretty sure the result is the same and whether you need to use `getDay` or `getUTCDay` is a display issue that affects both variants. I could see it being more convenient if you're just dumping the raw values without any transformation or you don't know what data type is stored in a co…

You're right that you'll have exactly the same issues if you use Date with a string.

However, with a string date you can bypass the Date class completely in many use cases.

My experience with this is anecdotal, but I've ran into many bugs due to conversion between day-precision values and Date objects. One example was a date picker library that returned values at 00:00 in the local timezone instead of UTC, which were then not correctly converted to UTC-based values before being persisted.

Once again, these are all preventable issues - just use UTC everywhere. I've just seen issues like that happen enough that I prefer never converting between dates and timestamps if I can avoid it.

Re: Show HN: Bi-directional sync between Postgres and SQLite

#75
post #5
post #4

Earlier quoted context omitted.

SQLite has a number of functions to work with time but the underlying storage format is just going to be--depending on what precision/range of timestamp you want--a float or an integer... to the extent, of course, that SQLite has types at all (as it frankly doesn't, thereby making this question kind of moot).

SQLite has five types, documented here: https://www.sqlite.org/datatype3.html null, integer, real, text, blob It has a historically cavalier attitude to enforcing them (which I believe it inherited from TCL) but that changed in November 2021 with the release of strict table mode in version 3.37.0: https://www.sqlite.org/stricttables.html

Ah! I mean, only two years old, but OK ;P. Regardless: my point stands that you can trivially store and work with timestamps in SQLite as they are simply a real or integer or text; like, the idea that you need a special timestamp data type or you can't work with such values in an engine that only two years ago got support for checking the types at all and encourages you to just write code is highly strange.

Re: Show HN: Bi-directional sync between Postgres and SQLite

#76

Sorry for being a noob but is this "tech" similar to the rails offline video linked or am I misunderstanding? I don't get why you would need a service instead of building it into the webapp/site? https://www.youtube.com/watch?v=Gj8ov0cOuA0

I just scanned through the video, but it seems to focus on persisting client-side changes locally when offline, which only covers one part of the bigger problem.

Difficulties quickly come in if you want to: 1. Persist large amounts of data on the client. 2. Keep the data in sync incrementally (too much data to re-download every time). 3. Keep the data up-to-date in realtime (streaming changes). 4. Keep the data consistent, especially across multiple tables / types.

Many "offline" solutions are actually closer to caching rather than offline-first, which introduce all the issues associated with cache invalidation.

Also, if you're just working with a single table, you might get by with just using updated_at timestamps and soft deletes to be able to get incremental changes from the server. Even then you need to be careful with consistency issues, e.g. making sure timestamps are always in order and without duplicates. And if you start adding more tables, the complexity quickly increases.

Re: Show HN: Bi-directional sync between Postgres and SQLite

#77

Sorry for being a noob but is this "tech" similar to the rails offline video linked or am I misunderstanding? I don't get why you would need a service instead of building it into the webapp/site? https://www.youtube.com/watch?v=Gj8ov0cOuA0

I just scanned through the video, but it seems to focus on persisting client-side changes locally when offline, which only covers one part of the bigger problem. Difficulties quickly come in if you want to: 1. Persist large amounts of data on the client. 2. Keep the data in sync incrementally (too much data to re-download every time). 3. Keep the data up-to-date in realtime (streaming changes). 4. Keep the data consi…

thank you for pointing that out, yes. I just saw the video uses indexedDB (probably suboptimal although mdn says its ok for storing complex data on the client) and also its just for 'bridging' no internet access when doing input on the device. thank you for taking the time to answer, I see a bit clearer now.

Re: Show HN: Bi-directional sync between Postgres and SQLite

#79

Earlier quoted context omitted.

I just scanned through the video, but it seems to focus on persisting client-side changes locally when offline, which only covers one part of the bigger problem. Difficulties quickly come in if you want to: 1. Persist large amounts of data on the client. 2. Keep the data in sync incrementally (too much data to re-download every time). 3. Keep the data up-to-date in realtime (streaming changes). 4. Keep the data consi…

thank you for pointing that out, yes. I just saw the video uses indexedDB (probably suboptimal although mdn says its ok for storing complex data on the client) and also its just for 'bridging' no internet access when doing input on the device. thank you for taking the time to answer, I see a bit clearer now.

IndexedDB is fine - just use some wrapped that makes it easier to work with. Our current SQLite implementation actually stores the underlying data inside IndexedDB - mostly due to a lack of better options (but that's busy changing with OPFS).

The difficult parts are mostly related to keeping the local data in sync with the server, whether that uses SQLite, IndexedDB or some other database.

Re: Show HN: Bi-directional sync between Postgres and SQLite

#80
post #48

Earlier quoted context omitted.

Yeah it's a shocker that even major apps like Apple's Weather aren't local first. Lots of room for improvement.

Why would you want a local first weather app? It's not very useful to have yesterday's weather or even from one hour ago.

Great question and that's what you'd think. I spend a lot of time in the outdoors though, and on day 3 of a camping trip into an area with no signal, I'd rather have a stale prediction for the day 4 weather than no prediction.
Post reply on HN