This is a great article about building global apps that require multi-region deployments. Thanks for sharing. Curious about the transaction retry errors for UPDATE that required 2 days to resolve. Probably could of been avoided using a distributed SQL database that supports a read committed isolation level ¯\_(ツ)_/¯ For those going down this path, maybe check out open source YugabyteDB. There is a great doc about how to build Global Apps using various application design patterns: https://docs.yugabyte.com/preview/develop/build-global-apps/
Blueprint for a distributed multi-region IAM with Go and CockroachDB
21–30 of 34 posts
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#22Good post, side remark our experience with kratos have been mixed while self hosting the solution. You can feel OSS is second class for them (lots of PR never getting merged, endless debates and little progress in code), it's OK its a business and they are not doing support contracts. Just know what you are getting into. Just my experience, might be different with other products.
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#23If you’d like to deploy a containers or even Ory itself to multi-region cloud, you should check out EdgeNode ( https://edgenode.com ), which I helped build
A friendly note: when I visited your site, I immediately clicked away when I saw that learning more about the deployment process, pricing, etc required me to sign up.
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#24Good post, side remark our experience with kratos have been mixed while self hosting the solution. You can feel OSS is second class for them (lots of PR never getting merged, endless debates and little progress in code), it's OK its a business and they are not doing support contracts. Just know what you are getting into. Just my experience, might be different with other products.
Hey Sytten,
I work in the community team for Ory.
OSS is actually very important for us and definitely not second class. (see also https://www.ory.sh/docs/open-source/commitment)
The assumption that Ory does not offer support contracts for self-hosted Ory is wrong (although we did not in the past, when the team was smaller).
We are doing contracts for companies using our software self-hosted: See here and contact us if you are interested! https://www.ory.sh/support/ This way we can assign engineers to your case and work on any issues you encounter or work on any contributions or features required.
Ory releases all features for free for everyone to use. What is not free however is our time and work. To merge a PR/add a new feature/etc. a significant amount of time is needed to make sure the code lives up to standards, passes all tests, any security implications, etc. This depends on the feature of course, but the one you are alluding to is probably one of those. See the Code of Conduct on OSS support as well: https://github.com/ory/hydra/blob/master/CODE_OF_CONDUCT.md
I hope that makes it clearer and feel free to reach out to me directly in the Ory Community on github or slack.
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#25Good post, side remark our experience with kratos have been mixed while self hosting the solution. You can feel OSS is second class for them (lots of PR never getting merged, endless debates and little progress in code), it's OK its a business and they are not doing support contracts. Just know what you are getting into. Just my experience, might be different with other products.
Sorry to hear that this has been your experience! What exactly was the issue for you? It’s true that there are lots of open PRs. We’re a small team and often busy with customer requirements which doesn’t allow us to get some community PRs over the finishing line (finish tests, refactor code, fix remaining bugs, do security reviews, …).
Sometimes, PRs are not aligning with an architecture or API principle which is when they often go stale. This is why we generally require design documents for changes or additions to APIs.
Saying that the open source is second class is a false accusation in my view:
- Over 1500 PRs merged in Ory Kratos alone: https://github.com/ory/kratos/pulls
- Very active contributor and commit frequency: https://github.com/ory/kratos/graphs/contributors?from=2018-...
- A growing community and footprint
Also, we do offer support contracts for self hosted environments - this is relatively new though: https://www.ory.dev/support/
It is true though that have to balance open source work and things that people pay us for. It’s the only way to ensure that Ory open source, for which we have a deep commitment, continues for a long time.
Hope this makes sense!
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#26I can only concur that multi-region apps are becoming the new normal.
Deploying app instances across distant locations was never an issue. However, databases used to be the bottleneck. I'm glad to see that changing, thanks to CockroachDB and YugabyteDB.
My favorite multi-region deployment mode is geo-partitioned deployment. This is when a database automatically pins user data to specific locations, ensuring low latency for both reads and writes, regardless of user location. One-minute demo how it works: https://www.youtube.com/watch?v=9ESTXEa9QZY&list=PL8Z3vt4qJT...
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#27This is a great article about building global apps that require multi-region deployments. Thanks for sharing. Curious about the transaction retry errors for UPDATE that required 2 days to resolve. Probably could of been avoided using a distributed SQL database that supports a read committed isolation level ¯\_(ツ)_/¯ For those going down this path, maybe check out open source YugabyteDB. There is a great doc about how…
Flexibility in isolation levels is coming in CRDB :)
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#28I suspect most business logic can handle 25ms for authz and that’s the right trade off. I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies? I work on an IAM system that is sub-ms p99 for our authz checks, with policies and keys pushed to each network edge instead of running a centralized system. The biggest perf hits are crypto verification and logging to the fs. We fai…
not sure if it applies but depending on instance type I usually see pings in the .55ms range in a single AZ in AWS, cross-AZ pings higher (implying it is hard to be sub ms for many types of durable applications, especially if disk/S3 is involved)
When discussing GCP, the latency between AZs within the same region is approximately 5 ms. Thus, if you have a 3-node database cluster spanning 3 AZs (all within the same region), a transaction can be committed in the 5-10ms range using Raft. CockroachDB should handle this seamlessly.
However, if you're considering a multi-region setup, the latency will depend on the distance between the regions.That's why usually you define a preferred region (that stores primary copy of the records) or deploy in a geo-partitioned mode (when data is automatically pinned to configured regions).
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#29Earlier quoted context omitted.
I’ve watched y’all’s Papers We Love talk about Zanzibar and have recommended authzed to organizations bootstrapping permission modeling. It’s been awhile, is the gist that Spanner’s coordinated clocks allow tighter consensus (i.e. faster writes) and caching provides read-my-write consistency?
Thanks for watching our presentation and recommending our solution. Unfortunately, nothing is ever simple; comparing Spanner and CockroachDB is comparing apples to oranges. Two years ago, we wrote an article that details exactly how the differences matter in terms of a Zanzibar implementation[0], but I can give as short of a summary as possible: Spanner is linearizable and CockroachDB only guarantees external consist…
I can attest to that statement: comparing Spanner and CockroachDB is difficult. Spanner and Fauna (where I work) are more comparable (Fauna is based on Calvin, see [0]) since they both support strict serializability (in different ways). The article referenced here is excellent, and it highlights what we've seen from some customers: CockroachDB is (to say the least) a challenge to learn and adequately deploy, I've seen a few others that have a similar lessons-learned result. I'm glad, however, that highly consistent distributed databases provide value in these implementations. Although not OSS, Fauna is comparable and more turnkey (read: much less ops) than these options.
[0]: https://fauna.com/blog/distributed-consistency-at-scale-span...
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#30If you’d like to deploy a containers or even Ory itself to multi-region cloud, you should check out EdgeNode ( https://edgenode.com ), which I helped build
A friendly note: when I visited your site, I immediately clicked away when I saw that learning more about the deployment process, pricing, etc required me to sign up.
Thanks for the feedback. This helps us a lot!
We’re currently in a early stage, but more info will be available to public in the next couple of weeks. You can fill out the form, if you want to be in the known: https://tally.so/r/w2ajRb
Thanks again, appreciate your honest feedback.