Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RxDB – a real-time database on top of PouchDB

github.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: RxDB – a real-time database on top of PouchDB

#11

Post by lukevp »

diegoperini wrote: post #6

Looks amazing. I always loved how Rx composes with IO on the client side, this looks like the missing half. I hope it survives. For people who are soon to be choosing a stack for their projects, please be careful. In 2015, we adopted RethinkDB which had very similar ambitions as RxDB and was open source. Unfortunately, RethinkDB is now abandoned (kinda). Many promising subscribe-able databases are still experimental.…

I agree, I think the move to a more mature state management on the client in general sort of obviated the needs for these databases. Eg if you’re using redux, or mobx, or Apollo, you can cache offline data directly in your state tree, and can define validation functions locally on state transitions to keep your data valid with your business rules while it’s offline. I still feel there’s a space for this tech, but that it needs to integrate into state management as a first class citizen.

Object definitions, graphs, and validation functions need to be definable in one place and replicated to clients. I’ve never seen this implemented, but if you don’t, you end up either having a schemaless DB or building your schema twice, and same with validation functions.

I shouldn’t have to create functions to observe changes and copy them to and from my redux state, or build my ui around PouchDB’s lifecycle methods. There are middleware providers out there but it’s not first class.

Re: RxDB – a real-time database on top of PouchDB

#12

Post by lukevp »

typingmonkey wrote: post #10
lukevp wrote: post #9

The concept or RxDB, being able to iterate observables of your change stream, is great. Centering on schemas and typescript is as well. The broadcast channel based leader election also solved the common issue with Pouch where you can’t respond to real-time changes and update your UI if a separate tab was watching too. It’s wise of them to also support pulling data from GraphQL. I built the first version of NoteBrook…

Actually, the long term plan is to move away from pouchdb. Before the last major release [1] I decided to make RxDB useable with other frontend/backend databases but then found out that I first should refactor some pain points. [1] https://github.com/pubkey/rxdb/issues/1636

That’s great to hear! Are you pubkey? If so you should put that in your profile!

I hope you know that I think RxDB is a great lib and going in a good direction! I recommended it to the Supabase team a couple weeks ago. I am less long PouchDB, it feels like an old generation of this tech and not the best way to solve these problems in 2020.

Re: RxDB – a real-time database on top of PouchDB

#13

Post by typingmonkey »

lukevp wrote: post #12
typingmonkey wrote: post #10

Earlier quoted context omitted.

Actually, the long term plan is to move away from pouchdb. Before the last major release [1] I decided to make RxDB useable with other frontend/backend databases but then found out that I first should refactor some pain points. [1] https://github.com/pubkey/rxdb/issues/1636

That’s great to hear! Are you pubkey? If so you should put that in your profile! I hope you know that I think RxDB is a great lib and going in a good direction! I recommended it to the Supabase team a couple weeks ago. I am less long PouchDB, it feels like an old generation of this tech and not the best way to solve these problems in 2020.

I updated my profile. I do not know supabase, is it similar to hasura? They spend quite some effort [1] to make the RxDB graphql replication working with their backend. Maybe supabase can use a similar wrapper over GraphQL.

[1] https://hasura.io/learn/graphql/react-rxdb-offline-first/int...

Re: RxDB – a real-time database on top of PouchDB

#14

Post by k__ »

typingmonkey wrote: post #10
lukevp wrote: post #9

The concept or RxDB, being able to iterate observables of your change stream, is great. Centering on schemas and typescript is as well. The broadcast channel based leader election also solved the common issue with Pouch where you can’t respond to real-time changes and update your UI if a separate tab was watching too. It’s wise of them to also support pulling data from GraphQL. I built the first version of NoteBrook…

Actually, the long term plan is to move away from pouchdb. Before the last major release [1] I decided to make RxDB useable with other frontend/backend databases but then found out that I first should refactor some pain points. [1] https://github.com/pubkey/rxdb/issues/1636

Ah, nice.

Does it work with AppSync? It seems to me that Amplify/DataStore is a product like RxDB, and they use AppSync as backend.

Edit: I just saw it seems to integrate custom GraphQL, so it probably a direct competitor to DataStore in that regard.

Re: RxDB – a real-time database on top of PouchDB

#15

Post by typingmonkey »

k__ wrote: post #14
typingmonkey wrote: post #10

Earlier quoted context omitted.

Actually, the long term plan is to move away from pouchdb. Before the last major release [1] I decided to make RxDB useable with other frontend/backend databases but then found out that I first should refactor some pain points. [1] https://github.com/pubkey/rxdb/issues/1636

Ah, nice. Does it work with AppSync? It seems to me that Amplify/DataStore is a product like RxDB, and they use AppSync as backend. Edit: I just saw it seems to integrate custom GraphQL, so it probably a direct competitor to DataStore in that regard.

No it does not work with appsync, but it could be hacked into doing that by modifiing the GraphQL plugin. There would be a huge value in replacing the aws datastore mainly because of its missing basic features like complex queries or queries with sort params.

Re: RxDB – a real-time database on top of PouchDB

#16

Post by k__ »

typingmonkey wrote: post #15
k__ wrote: post #14

Earlier quoted context omitted.

Ah, nice. Does it work with AppSync? It seems to me that Amplify/DataStore is a product like RxDB, and they use AppSync as backend. Edit: I just saw it seems to integrate custom GraphQL, so it probably a direct competitor to DataStore in that regard.

No it does not work with appsync, but it could be hacked into doing that by modifiing the GraphQL plugin. There would be a huge value in replacing the aws datastore mainly because of its missing basic features like complex queries or queries with sort params.

Ah good to know you already looked into that :D

Thanks for the info!

Re: RxDB – a real-time database on top of PouchDB

#17

Post by typingmonkey »

k__ wrote: post #16
typingmonkey wrote: post #15

Earlier quoted context omitted.

No it does not work with appsync, but it could be hacked into doing that by modifiing the GraphQL plugin. There would be a huge value in replacing the aws datastore mainly because of its missing basic features like complex queries or queries with sort params.

Ah good to know you already looked into that :D Thanks for the info!

Yes I did :D . I wrote a full master thesis about the differences and tradeoffs of these "realtime" databases.

Re: RxDB – a real-time database on top of PouchDB

#18

Post by k__ »

typingmonkey wrote: post #17
k__ wrote: post #16

Earlier quoted context omitted.

Ah good to know you already looked into that :D Thanks for the info!

Yes I did :D . I wrote a full master thesis about the differences and tradeoffs of these "realtime" databases.

Well, DataStore is rather new AND AWS specific, so I simply assumed you wouldn't even care.

but good to know. Reminds me of the fact that I still have to write mine, lol

Re: RxDB – a real-time database on top of PouchDB

#19

Post by remon »

How is this different from Firebase RTDB? And perhaps more importantly, does it address the scalability and consistency issues associated with Firebase RTDB? Google introduced Firestore to Firebase specifically because RTDB has limited usability for larger real world applications that go beyond "sync device state to DB".

Even the offline first paradigm is fundamentally flawed in general and certainly when it comes to offline data manipulation. Either you can afford to mutate your data on the local device and sync it when possible, in which case you clearly don't need subscriptions to real-time mutations of remote data because within that scope you are the source of truth. Or, you're interested in mutations of real-time data from multiple clients in which case you need to deal with conflict resolution (mutually exclusive changes of data) which is not reliably possible with this model and scalability (linear increase in pub/sub * increasing query cost = exponential scaling).

Are there any large projects or companies that currently have this in production?

Re: RxDB – a real-time database on top of PouchDB

#20

Post by remon »

diegoperini wrote: post #6

Looks amazing. I always loved how Rx composes with IO on the client side, this looks like the missing half. I hope it survives. For people who are soon to be choosing a stack for their projects, please be careful. In 2015, we adopted RethinkDB which had very similar ambitions as RxDB and was open source. Unfortunately, RethinkDB is now abandoned (kinda). Many promising subscribe-able databases are still experimental.…

Subscribe-on-update databases are, almost by definition, problematic to use at scale as a generic storage solution. The fundamental problem is that they do not solve many real world problems efficiently enough to warrant the significantly higher running cost. Of course there are exceptions but it'll be hard to launch a MongoDB type of product that uses a subscription only model (see Firebase RTDB and its problems and lack of adoption in this space)

The reason developers gravitate towards subscription/rx based paradigms is because it results in very clean architecture and code. Unfortunately it comes at a cost which increases rather than decreases per client/user when volume increases. Some companies or projects can absorb that cost but not all, and typically less so when the project or its userbase grows.

A subscription based model will do work whenever data changes for each subscriber whereas more traditional pull based architectures only do work when a client specifically needs the data. This can be mitigated to some extent by being micromanaging subscriptions but that kills most of the value of the model.

There are also plenty of issues with this model if multiple clients are allowed to write to the same data which every single example project seems to try and do. There's a reason master-master updates, consensus algorithms and CRDTs all come at significant cost. It's usually hard. And when it's easy you probably don't need it subscribe-on-update in the first place.

Post reply on HN