Blueprint for a distributed multi-region IAM with Go and CockroachDB
1–10 of 34 posts
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#2This isn't the typical 1000 word, "here's how we did it, now use our thing" company fluff blog post. What a great writeup. Sometimes reading docs, it is hard to figure out the fine details when making a decision. Your comparison of Regional Tables, Regional By Row Tables, and Global Tables is a really nice summary of the pros & cons of each. Well done.
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#3One of the reasons I started writing Marmot (https://maxpert.github.io/marmot/) was for replicating bunch of tables across regions that were read heavy. I even used it for cache replication (because who cares if it’s a cache miss, but a hit will save me time and money). It’s hard to make such blue prints in early days of product, and by the time you hit a true growth almost everyone builds a custom solution for multi-region IAM.
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#4I suspect most business logic can handle 25ms for authz and that’s the right trade off. I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies?
I work on an IAM system that is sub-ms p99 for our authz checks, with policies and keys pushed to each network edge instead of running a centralized system. The biggest perf hits are crypto verification and logging to the fs. We fail-closed to last known policy state when we have partitions, data loss would imply the application service or datastore proxy is lost. We measure policy deploy times in minutes though, and it’s eventually consistent.
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#5Awesome post, really. One of the best I've read in a while!
Total side question, if anyone knows -- what tool (if any?) was used for the graphics in this article? The dot matrix looking map style stuff? I really dig it.
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#6I suspect most business logic can handle 25ms for authz and that’s the right trade off. I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies? I work on an IAM system that is sub-ms p99 for our authz checks, with policies and keys pushed to each network edge instead of running a centralized system. The biggest perf hits are crypto verification and logging to the fs. We fai…
not sure if it applies but depending on instance type I usually see pings in the .55ms range in a single AZ in AWS, cross-AZ pings higher (implying it is hard to be sub ms for many types of durable applications, especially if disk/S3 is involved)
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#7I suspect most business logic can handle 25ms for authz and that’s the right trade off. I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies? I work on an IAM system that is sub-ms p99 for our authz checks, with policies and keys pushed to each network edge instead of running a centralized system. The biggest perf hits are crypto verification and logging to the fs. We fai…
>I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies?
That's correct. In a Zanzibar-like model, you have a global storage, but individual clusters in each datacenter/edge providing consistency-aware caching. This means p99 can be something like 25ms, but p95 or p50 is often FAR lower.
Disclosure: I'm a co-creator and maintainer of SpiceDB[0]
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#8Awesome post, really. One of the best I've read in a while! Total side question, if anyone knows -- what tool (if any?) was used for the graphics in this article? The dot matrix looking map style stuff? I really dig it.
Thank you! I appreciate that a lot!
Our designers will love that feedback! Unfortunately it’s not a shelf product but they used Figma to design the graphs.
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#9I suspect most business logic can handle 25ms for authz and that’s the right trade off. I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies? I work on an IAM system that is sub-ms p99 for our authz checks, with policies and keys pushed to each network edge instead of running a centralized system. The biggest perf hits are crypto verification and logging to the fs. We fai…
One of Ory’s core competencies is permissions. We built the first Google Zanzibar implementation in the world and it’s part of Ory Network‘s global multi-region platform (https://github.com/ory/keto)
A push model is also valid if you’re heavy on policies and can accept eventual consistency. We will investigate how to generally push things to the edge (like we did with Ory Edge Sessions) or to cryptographic verification wherever staleness is acceptable.
By solving the primitives correctly in the beginning (with a multi region architecture) that job does become a lot easier, which is what we decided doing at Ory :)
Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB
#10This isn't the typical 1000 word, "here's how we did it, now use our thing" company fluff blog post. What a great writeup. Sometimes reading docs, it is hard to figure out the fine details when making a decision. Your comparison of Regional Tables, Regional By Row Tables, and Global Tables is a really nice summary of the pros & cons of each. Well done.
Thank you, I appreciate that feedback because this was the explicit goal of writing the article: Informing that multi region is no longer just a vision for companies that aren’t Google; Sharing how difficult it is; And some of the learnings made along the way!
Personally, I am extremely proud of the work. I believe that in a year or two, most companies will adopt multi region IAM (hopefully from Ory as we’re currently the only ones capable of this). :)
And what could be better than hearing these kind words from the critical readers on HN :)
Cheers!