Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Blueprint for a distributed multi-region IAM with Go and CockroachDB

ory.dev

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB

#11

Post by aeneas_ory »

maxpert wrote: post #3

One of the reasons I started writing Marmot ( https://maxpert.github.io/marmot/ ) was for replicating bunch of tables across regions that were read heavy. I even used it for cache replication (because who cares if it’s a cache miss, but a hit will save me time and money). It’s hard to make such blue prints in early days of product, and by the time you hit a true growth almost everyone builds a custom solution for mul…

That is true! And the reason why we decided to build this and offer it to everyone - building multi-region IAM is incredibly difficult and expensive and typically not the core competency of an average software company.

Also, very interesting project. I love SQLite and what the community is contributing to it - yours included!

Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB

#13

Post by leoqa »

aeneas_ory wrote: post #9
leoqa wrote: post #4

I suspect most business logic can handle 25ms for authz and that’s the right trade off. I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies? I work on an IAM system that is sub-ms p99 for our authz checks, with policies and keys pushed to each network edge instead of running a centralized system. The biggest perf hits are crypto verification and logging to the fs. We fai…

One of Ory’s core competencies is permissions. We built the first Google Zanzibar implementation in the world and it’s part of Ory Network‘s global multi-region platform ( https://github.com/ory/keto ) A push model is also valid if you’re heavy on policies and can accept eventual consistency. We will investigate how to generally push things to the edge (like we did with Ory Edge Sessions) or to cryptographic verifica…

My intuition is that offering this as a service you’re targeting business logic that can handle 25ms authz. We’re on the core path in a latency sensitive industry, and end up running many permissions checks at various layers for a single api call.

Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB

#14

Post by aeneas_ory »

leoqa wrote: post #13
aeneas_ory wrote: post #9

Earlier quoted context omitted.

One of Ory’s core competencies is permissions. We built the first Google Zanzibar implementation in the world and it’s part of Ory Network‘s global multi-region platform ( https://github.com/ory/keto ) A push model is also valid if you’re heavy on policies and can accept eventual consistency. We will investigate how to generally push things to the edge (like we did with Ory Edge Sessions) or to cryptographic verifica…

My intuition is that offering this as a service you’re targeting business logic that can handle 25ms authz. We’re on the core path in a latency sensitive industry, and end up running many permissions checks at various layers for a single api call.

Absolutely, having P99 of sub-ms is of course way more attractive than 25ms - with a SaaS offering you always have the network latency to the provider in the path, which is why multi region capabilities are so important for this case. But you’ll never beat systems where the decision can be made locally.

Have you any documentation on your approach publicly available? I‘d love to get some education and insights from other large scale authz systems! We have a couple of ideas such as running a local replica in our customer’s stack but nothing concrete yet.

Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB

#15

Post by leoqa »

jzelinskie wrote: post #7
leoqa wrote: post #4

I suspect most business logic can handle 25ms for authz and that’s the right trade off. I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies? I work on an IAM system that is sub-ms p99 for our authz checks, with policies and keys pushed to each network edge instead of running a centralized system. The biggest perf hits are crypto verification and logging to the fs. We fai…

>I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies? That's correct. In a Zanzibar-like model, you have a global storage, but individual clusters in each datacenter/edge providing consistency-aware caching. This means p99 can be something like 25ms, but p95 or p50 is often FAR lower. Disclosure: I'm a co-creator and maintainer of SpiceDB[0] [0]: https://github.com/authz…

I’ve watched y’all’s Papers We Love talk about Zanzibar and have recommended authzed to organizations bootstrapping permission modeling.

It’s been awhile, is the gist that Spanner’s coordinated clocks allow tighter consensus (i.e. faster writes) and caching provides read-my-write consistency?

Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB

#16

Post by aeneas_ory »

endisneigh wrote: post #12

Seems that cockroachdb saved the day here with its multi region capabilities. Did the other vendors have the same capabilities? Specifically things like the regional tables and columns.

We made the decision to choose Cockroach in 2018 and back then no product had these capabilities. We stuck to CRDB because they delivered on their product vision and as far as our research went they have the most advanced solution.

Even Google Cloud Spanner (NOT the same as Google Spanner - the internal DB) lacks a couple of things we needed for data homing.

Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB

#17

Post by jzelinskie »

leoqa wrote: post #15
jzelinskie wrote: post #7

Earlier quoted context omitted.

>I think Google’s Zanzibar is also centralized but leverages extreme caching to get lower latencies? That's correct. In a Zanzibar-like model, you have a global storage, but individual clusters in each datacenter/edge providing consistency-aware caching. This means p99 can be something like 25ms, but p95 or p50 is often FAR lower. Disclosure: I'm a co-creator and maintainer of SpiceDB[0] [0]: https://github.com/authz…

I’ve watched y’all’s Papers We Love talk about Zanzibar and have recommended authzed to organizations bootstrapping permission modeling. It’s been awhile, is the gist that Spanner’s coordinated clocks allow tighter consensus (i.e. faster writes) and caching provides read-my-write consistency?

Thanks for watching our presentation and recommending our solution.

Unfortunately, nothing is ever simple; comparing Spanner and CockroachDB is comparing apples to oranges. Two years ago, we wrote an article that details exactly how the differences matter in terms of a Zanzibar implementation[0], but I can give as short of a summary as possible: Spanner is linearizable and CockroachDB only guarantees external consistency for transactions that share rows. The post outlines how we workaround this and we've also more recently talked about how we've managed to scale that to 1M requests per second[1]. Our team focuses a lot on CockroachDB because we offer a permission systems that can span not only regions within a single cloud, but across various cloud providers. However, if you're all in on GCP, SpiceDB itself supports Cloud Spanner (which we also use in production for our GCP-only customers).

[0]: https://authzed.com/blog/prevent-newenemy-cockroachdb

[1]: https://authzed.com/blog/maximizing-cockroachdb-performance

Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB

#18

Post by leoqa »

aeneas_ory wrote: post #14
leoqa wrote: post #13

Earlier quoted context omitted.

My intuition is that offering this as a service you’re targeting business logic that can handle 25ms authz. We’re on the core path in a latency sensitive industry, and end up running many permissions checks at various layers for a single api call.

Absolutely, having P99 of sub-ms is of course way more attractive than 25ms - with a SaaS offering you always have the network latency to the provider in the path, which is why multi region capabilities are so important for this case. But you’ll never beat systems where the decision can be made locally. Have you any documentation on your approach publicly available? I‘d love to get some education and insights from ot…

It’s quite similar mechanically to this blog post about Uber’s policy framework: https://www.uber.com/blog/attribute-based-access-control-at-....

We have an additional scaling dimension though, as our permission model is richer and mutable by end users, therefore our policies are not uniform. For special hot-path services, we use symmetric keys to reduce latency further but that makes rotations complicated.

Re: Blueprint for a distributed multi-region IAM with Go and CockroachDB

#20

Post by aeneas_ory »

leoqa wrote: post #18
aeneas_ory wrote: post #14

Earlier quoted context omitted.

Absolutely, having P99 of sub-ms is of course way more attractive than 25ms - with a SaaS offering you always have the network latency to the provider in the path, which is why multi region capabilities are so important for this case. But you’ll never beat systems where the decision can be made locally. Have you any documentation on your approach publicly available? I‘d love to get some education and insights from ot…

It’s quite similar mechanically to this blog post about Uber’s policy framework: https://www.uber.com/blog/attribute-based-access-control-at-... . We have an additional scaling dimension though, as our permission model is richer and mutable by end users, therefore our policies are not uniform. For special hot-path services, we use symmetric keys to reduce latency further but that makes rotations complicated.

Awesome, thank you for following up! Will give this a read before bed. Would love to understand the encryption pieces, as I very much get the need for frequent updates of permissions (typically append, only sometimes remove). If you ever happen to blog about it please let me know :)
Post reply on HN