111–120 of 137 posts
Why is it called Zanzibar though? I'm kind of intrigued
There's another thread about naming it - https://twitter.com/LeaKissner/status/1136691523104280576 The original name was Spice, which was nixed from a higher-up; they went to Zanzibar, one of the Spice Islands.
Earlier quoted context omitted.
It's just a saying. All we do is move protos from one service to another. JSON is definitely not the right stuff.
The encoding/decoding cost is painful :( I mean in this context if you're doing that level of scale. For a lot of purposes json is totally fine.
Earlier quoted context omitted.
Throwing servers at the problem is less impressive then thinking very hard and solving it with less.
If your conclusion is "throwing servers at problems" after years of reading the papers about Google infrastructure, you probably are a non infrastructure guy. A serious conclusion should be that all these infrastructure enable application devs and researchers alike "to throw servers at problem". And these work are exactly the opposite, where they spent years and sometimes even decades meditating the nifty and figure…
Excellent paper. As someone who has worked with filesystems and ACLs, but never touched Spanner before, I have some questions for any Googler who has played with Zanzibar. (in part because full-on client systems examples are limited) A check my understanding: Zanzibar is being optimized to handle zookies that are a bit stale (say 10s) old. In this case, the indexing systems (such as Leapord) can be used to vastly acc…
> 3. There's nothing Zanzibar can do but "pointer chase", resulting in a large number of serial ACL checks?
Zanzibar enforced a max depth and would fail if the pointer-chasing traversed too deeply. Zanzibar would also fail if it traversed too many nodes.
> 4. How do clients consistently update ACLs alongside their "reverse edges"?
One of the recommended solutions was to store your full ACL (which includes a Zookie) in the same Spanner row of whatever it protected. So, if your ACL is for books, you might have:
CREATE TABLE books (
book SERIAL PRIMARY KEY
acl ZanzibarAcl
);
1. Fetch Zookie from Spanner
2. Call Zanzibar.Check with the zookie
> but how do I consistently view which groups a user is a member of?
I remember this as a large source of pain to implement. Zanzibar didn't support this use-case directly. As rpang mentioned in a sibling comment, you need an ACL-aware index. Essentially, the algorithm is:
1. Call Zanzibar.Check on all groups the user might be a part of.
There's a bunch of clever tricks you can use to prune the search space that I don't know the details of.
Earlier quoted context omitted.
This isn't a login service, this is an ACL service. Related space, but different concerns. You wouldn't send a user's password here to find out if it's correct (authentication), you'd use this to figure out if a user can do something once you know who they are (authorization) :) Also, generating the login page etc is often more expensive than the actual 'validate the username and password'. Getting to the server is a…
Awkward. I realized it was used for authz, but for some reason I assumed it would be used for authn as well. Now I’m wondering how Google does authn... And yeah, the second half of my comment is trying to scope down the comparison to one that is reasonably “fair”
That's my corner of Google. We haven't published anything comparable to this paper in the time I've worked on it (maybe we could—I'm pleasantly surprised to see the Zanzibar folks got approval to share qps numbers and everything) but here's a bit about how it worked back in 2006:
https://www.usenix.org/legacy/event/worlds06/tech/prelim_pap...
Some of that still applies.
fwiw, while we do our fair share of password checking, we do a _lot_ more oauth token and cookie checking. Most folks just stay signed in on both mobile and web, so no need to recheck their passwords. In contrast, session credentials get checked on every request.
The distinguishing feature I see compared to other systems is the ACL ordering and consistency, which is indeed difficult to do at scale. Looks like Spanner is doing most the heavy lifting, great use case for the database.
If you don't have negative ACL entries then ordering is not important.
(You might be thinking of ordering ACEs in the ACL which isn't even a concept in Zanzibar)
The distinguishing feature I see compared to other systems is the ACL ordering and consistency, which is indeed difficult to do at scale. Looks like Spanner is doing most the heavy lifting, great use case for the database.
Using Windows style ACEs for ACLs is also perfectly scalable and consistent, (and more performant) so long as users don't end up in too many groups and objects only inherit ACLs from objects on the same shard. It's just no where as generalizable as Zanzibar which allows much more complex dependencies.
There's always tradeoffs! But this is the best system I've seen for the general ACL evaluation against non recently updated objects.
Earlier quoted context omitted.
99 - 99.9 - That is amazing.
I'm not saying this is the case at all, but I've noticed through experience that depending on how a system at scale is distributed that .1% outside your 99.9% may be impacting a specific user or group of users, or group of resources, or etc. So they may be getting 100% of their requests outside your 99.9% latency. Something interesting to think about.
Zanzibar almost certainly has this type of behavior. More complex ACL structures under recent evaluations perform worse.
I'd love to see percentile data for depth of operation, but clearly the paper is limited in content size.
Am I alone in thinking that 99.999% measured availability for a service so completely in the critical path for almost everything is relatively low? Phrased another way, when it is not availability, do end users experience service disruption, and if not, how is that mitigated?
You may have missed how they're defining it: > We define availability as the fraction of “qualified” RPCs the service answers successfully within latency thresholds: 5 seconds for a Safe request, and 15 seconds for a Recent request as leader re-election in Spanner may take up to 10 seconds. ... To compute availability, we aggregate success ratios over 90-day windows averaged across clusters. Figure 5 shows Zanzibar’s…
Earlier quoted context omitted.
There's another thread about naming it - https://twitter.com/LeaKissner/status/1136691523104280576 The original name was Spice, which was nixed from a higher-up; they went to Zanzibar, one of the Spice Islands.
Odd. Zanzibar is off the coast of Tanzania in East Africa. The Spice Islands (the Moluccas) are in Eastern Indonesia.
> the Zanzibar Archipelago, together with Tanzania's Mafia Island, are sometimes referred to locally as the "Spice Islands" (a term borrowed from the Maluku Islands of Indonesia).