MiniCouchDB in Rust • Hacker News Board

Re: MiniCouchDB in Rust

#21

Post by karmelapple » Sun, Jun 07, 2020, 5:41 PM UTC

tgtweak wrote: post #5

garrensmith wrote: post #3

Earlier quoted context omitted.

Thanks for reading. We have some plans around this, see the RFC - https://github.com/apache/couchdb-documentation/pull/424 In general, we are trying to move away from running your whole application in CouchDB. We would prefer that you have an application layer in front of your CouchDB instance. We recommend that you put up a proxy if you want to expose your replication to PouchDB. That way you can add security around…

In general the fewer technologies you need to bring into the mix the less failure domains you need to consider. Let's also add that having an option to provide this natively allows for safe/sane defaults that prevent someone from botching a reverse proxy authentication setup which is by no means trivial to your application developer. I can't see someone making a meaningful document level ACL outside of the db without…

We moved to CouchDB because we didn’t want to reinvent syncing. We wanted a tech that could handle the online and offline case well, and CouchDB and some supporting libraries (PouchDB, some native mobile OS libraries) helped with this.

Syncing was a solved problem in the CouchDB world, so we moved to CouchDB.

Access controls also seem like they should be a “solved problem,” something we can adopt that works well with CouchDB.

Is there some plan for a standard ACL approach for CouchDB users? If we need to put it into some app in front of the database, that eliminates the “solved” nature of sync... or at least introduces added complexities like a reverse proxy.

I really hope CouchDB could bring in some ACL management into the ecosystem as a standard, drop-in sort of approach, and not something that requires custom development for every single app that wants to use CouchDB / Cloudant.

Re: MiniCouchDB in Rust

#22

Post by karmelapple » Sun, Jun 07, 2020, 5:52 PM UTC

garrensmith wrote: post #3

lukevp wrote: post #2

This is really cool, thanks for your work! I really love couchdb. I have seen tickets for document based ACLs on their github recently, I think this is the last feature they need to open up usage across many more domains. The core app works great for replication between dbs and to the edge using pouchdb, but almost everyone is turned off by the database-per-user that is required to use it. If they solve that I think…

Thanks for reading. We have some plans around this, see the RFC - https://github.com/apache/couchdb-documentation/pull/424 In general, we are trying to move away from running your whole application in CouchDB. We would prefer that you have an application layer in front of your CouchDB instance. We recommend that you put up a proxy if you want to expose your replication to PouchDB. That way you can add security around…

I just read this more carefully, and my gut reaction is disappointment, but hopefully you can clarify if I’m misunderstanding anything.

Right at the top it says “Make the db-per-user pattern obsolete.”

Db-per-user enables trivial syncing between multiple devices for a single user. That means someone’s tablet, phone, and laptop can all stay in sync thanks to CouchDB’s replication. And we can do it without building anything extra, as long as we have a client library that talks CouchDB’s http API.

Does this new approach eliminate that possibility? Will we now need a server in front of the database for that?

Re: MiniCouchDB in Rust

#23

Post by xrd » Sun, Jun 07, 2020, 6:08 PM UTC

lukevp wrote: post #2

This is really cool, thanks for your work! I really love couchdb. I have seen tickets for document based ACLs on their github recently, I think this is the last feature they need to open up usage across many more domains. The core app works great for replication between dbs and to the edge using pouchdb, but almost everyone is turned off by the database-per-user that is required to use it. If they solve that I think…

This is interesting to me.

One of the things that really turned me off from firebase is the rules system where you have to build an ACL there. It's been a few years since I've played heavily with it, but I really disliked trying to essentially build my authorization system into a backend which I couldn't run locally and was primarily edited in their web console.

For me, writing a proxy that sits in front of couchdb is very simple. I use JWT tokens that get passed between components in the system. I keep my authorization logic out of couch. I can write normal unit tests on my proxy. And my pouchdb client code mirrors the structure of my backend structure, which is a mental model I really prefer.

I guess I'm saying I think having a separate database for each user makes more sense to me, not less.

The only thing I'm struggling with is running code when a document is updated. I have a polling client that watches for the _global_changes updates, but it seems really hacky. I wish there was a better way to get access to all database changes that looked like firestore functions.

Re: MiniCouchDB in Rust

#24

Post by lukevp » Sun, Jun 07, 2020, 6:10 PM UTC

karmelapple wrote: post #22

garrensmith wrote: post #3

Earlier quoted context omitted.

Thanks for reading. We have some plans around this, see the RFC - https://github.com/apache/couchdb-documentation/pull/424 In general, we are trying to move away from running your whole application in CouchDB. We would prefer that you have an application layer in front of your CouchDB instance. We recommend that you put up a proxy if you want to expose your replication to PouchDB. That way you can add security around…

I just read this more carefully, and my gut reaction is disappointment, but hopefully you can clarify if I’m misunderstanding anything. Right at the top it says “Make the db-per-user pattern obsolete.” Db-per-user enables trivial syncing between multiple devices for a single user. That means someone’s tablet, phone, and laptop can all stay in sync thanks to CouchDB’s replication. And we can do it without building any…

No, the problem right now is couchdb has no ACLs at the document level so a common pattern is to use a separate DB per user. If there are document level ACLs, you will still have to create a user in couchdb per user but can now use a single database and assign the permissions at the document level. You could continue using the current pattern if you wanted to. This new architecture has HUGE ramifications though because it enables easy ways to share / make public data just by changing ACLs, and it drastically simplifies backup and restore as well as multi master replication because a super-user (admin) can be used server side to sync everything. Honestly this 1 change should have been a main priority since the inception of couchdb because it further expands the key differentiator of couchdb, which as you said is to enable endpoint syncing for a subset of the overall documents down to end user devices for more performant access, creation, and offline use. This is assuming there is a way to default ACLs on documents to the creating user or role and not wide open. The wide open by default nature of couchdb was a major misstep and why they disabled “admin party” in 3.0.

Re: MiniCouchDB in Rust

#25

Post by asjw » Sun, Jun 07, 2020, 6:57 PM UTC

aswanson wrote: post #8

If you think about it, all future database development should be written in Rust.

Because the other several DBMS not written in Rust have shown what problem exactly?

Re: MiniCouchDB in Rust

#26

Post by lukevp » Sun, Jun 07, 2020, 7:00 PM UTC

daleharvey wrote: post #20

kungfooguru wrote: post #16

Earlier quoted context omitted.

Curious why they suggest a reverse proxy. It may be a holdover from a time before Erlang SSL got so many improvements? Heroku moved SSL termination to Erlang from ELB's and saw great improvements, plus they released a useful lib https://github.com/heroku/snit I suppose it could be related to their use of mochiweb still for the web layer. Maybe they'll add on HTTP/2 eventually and no longer recommend a reverse proxy.…

This is an architecture I use a lot. The reverse proxy is there to implement the replication, having your application understand the replication protocol is a big ask. Instead, your app takes http requests, handles the ones it is supposed to, and forwards database requests directly to (c|p)ouchdb (after checking auth) https://github.com/daleharvey/noted/blob/master/index.js is a very simple example of how it can work

Right, but if your application is just a straight proxy that’s stripping out a bearer jwt or something to validate, you can’t control which documents are being synced. So you have to have 1) some understanding of the underlying protocol so you can parse the request and reject it based on user if that user does not have read/write access to the db, or 2) replicate your authorization code into couchdb with users and db permissions, at which point you have the same db per user issue (or lack thereof) plus you now have 2 user stores and 2 layers. Once this gets sufficient complexity and you try to architect something non trivial (eg. User a owns write access to document b but shares read only access to document b to user C) you end up praying to god you built the protocol parsing just right or you might accidentally let users arbitrarily write to each other’s documents.

I acknowledge that this is the same issue in a traditional dbms, but it doesn’t try to handwave away this complexity from you. It doesn’t show examples on their site about how easy replication is to set up only to be betrayed later After you’ve already integrated and marveled at the couchdb sync performance when you can’t build real permissions and would’ve been better off using Postgres and getting a bigger community and ACID/CP. and let’s be honest, almost every app is gonna need a rdbms at least for transactional data anyway (we know you’re not storing stripe billing records in your couchdb) so then the question becomes, is the sync protocol even good enough anymore to warrant using couchdb?

Re: MiniCouchDB in Rust

#27

Post by Dowwie » Sun, Jun 07, 2020, 7:07 PM UTC

The 'mini-redis' project wasn't intended to build a redis clone as much as it was to teach many of the asyncio concepts in Rust with an example that uses an extensive amount of code comments. Such examples have been missing from the Rust ecosystem.

If you were inspired by mini-redis, why not go the extra mile and turn it into another great educational resource?

Re: MiniCouchDB in Rust

#28

Post by lukevp » Sun, Jun 07, 2020, 7:16 PM UTC

daleharvey wrote: post #20

kungfooguru wrote: post #16

Earlier quoted context omitted.

Curious why they suggest a reverse proxy. It may be a holdover from a time before Erlang SSL got so many improvements? Heroku moved SSL termination to Erlang from ELB's and saw great improvements, plus they released a useful lib https://github.com/heroku/snit I suppose it could be related to their use of mochiweb still for the web layer. Maybe they'll add on HTTP/2 eventually and no longer recommend a reverse proxy.…

This is an architecture I use a lot. The reverse proxy is there to implement the replication, having your application understand the replication protocol is a big ask. Instead, your app takes http requests, handles the ones it is supposed to, and forwards database requests directly to (c|p)ouchdb (after checking auth) https://github.com/daleharvey/noted/blob/master/index.js is a very simple example of how it can work

We really shouldn't be writing our own authentication layers (or anything security-related for that matter) unless necessary.

Case in point, this example code has a massive security issue that allows anyone to impersonate without tokens if there is an active authentication request open. Hopefully no one has used this example code to build a production system that has real user data.

This is a perfect example of why this should really be part of CouchDB/PouchDB itself and not something each person must write themselves. This should be solved once, solved right, vetted by the community, and be easy to fall into a pit of success.

I really like CouchDB and PouchDB as a product, but this insistence that this is the right path is really holding you guys back.

Re: MiniCouchDB in Rust

#29

Post by nullwarp » Sun, Jun 07, 2020, 7:17 PM UTC

This site is completely unusable without javascript, just spits out "You need to enable JavaScript to run this app."

Re: MiniCouchDB in Rust

#30

Post by lukevp » Sun, Jun 07, 2020, 7:34 PM UTC

karmelapple wrote: post #21

tgtweak wrote: post #5

Earlier quoted context omitted.

In general the fewer technologies you need to bring into the mix the less failure domains you need to consider. Let's also add that having an option to provide this natively allows for safe/sane defaults that prevent someone from botching a reverse proxy authentication setup which is by no means trivial to your application developer. I can't see someone making a meaningful document level ACL outside of the db without…

We moved to CouchDB because we didn’t want to reinvent syncing. We wanted a tech that could handle the online and offline case well, and CouchDB and some supporting libraries (PouchDB, some native mobile OS libraries) helped with this. Syncing was a solved problem in the CouchDB world, so we moved to CouchDB. Access controls also seem like they should be a “solved problem,” something we can adopt that works well with…

That RFC/PR adds ACLs at the document level to CouchDB, but it has been open for a year now. I hope they merge it as well. Until that PR the party line has always been to do ACL outside the DB like you would with any other DB, which really doesn’t make much sense to me.