> There's also a story behind that project name. That is not the original project name. The original project name was force-removed by my SVP. Once my hands are free again, I can explain https://mobile.twitter.com/LeaKissner/status/113663143751427...
Zanzibar: Consistent, Global Authorization System
21–30 of 137 posts
Re: Zanzibar: Consistent, Global Authorization System
#22I love reading about Google's systems, but I wish I could work on those problems at scale, that is my dream really. I wonder what more systems Google has that we don't know about. I know Borg has become what we know as k8s but surely there must be more things that Google has made internally that are not open source. Curious about this and would like to know more about it from anyone in the trenches at Google.
On the other hand Google engineers are vendor-locked-in using Google specific tech one way or another.
Not strictly true. Most software would probably require at least some modifications to run internally but as far as I know there’s no policy preventing open source software in production, quite the opposite.
For example, here’s some information about memcache: https://www.quora.com/Does-Google-use-memcached-or-does-it-u...
There’s more, but if I can’t find a reference to them on Google Search I’ll assume its not in my place to discuss it publicly.
Using protobufs as a base layer may seem like lock-in, but it very much is the opposite. Protobufs are surprisingly simple and maybe even elegant once you get past the ugly parts, but most importantly it decouples software from arbitrary protocols and makes it much easier to deal with changing implementations. (Not to mention the potential for rich backwards and forwards compatibility.)
Re: Zanzibar: Consistent, Global Authorization System
#23Am I alone in thinking that 99.999% measured availability for a service so completely in the critical path for almost everything is relatively low? Phrased another way, when it is not availability, do end users experience service disruption, and if not, how is that mitigated?
You might not be alone but five nines is pretty good.
I seen many internal facing teams across many companies have SLOs of four nines or less. Five is pretty rare.
Re: Zanzibar: Consistent, Global Authorization System
#24“Zanzibar scales to trillions of access control lists and millions of authorization requests per second to support services used by billions of people. It has maintained 95th-percentile latency of less than 10 milliseconds and availability of greater than 99.999% over 3 years of production use” Impressive!
"This caching, along with aggressive pooling of read requests, allows Zanzibar to issue only 20 million read RPCs per second to Spanner." ("Only")
I'm surprised by all the numbers they give out: latency, regions, operation counts, even servers. The typical Google paper omits numbers on the Y axis of its most interesting graphs. Or it says "more than a billion", which makes people think "2B", when the actual number might be closer to 10B or even higher.
Re: Zanzibar: Consistent, Global Authorization System
#25Am I alone in thinking that 99.999% measured availability for a service so completely in the critical path for almost everything is relatively low? Phrased another way, when it is not availability, do end users experience service disruption, and if not, how is that mitigated?
I think you might be alone. 5.26 minutes of down time per year is beyond excellent for any moderately complex system.
Re: Zanzibar: Consistent, Global Authorization System
#26Earlier quoted context omitted.
The harsh truth of working at Google is that in the end you are moving protobufs from one place to another. They have the most talented people in the world but those people still have to do some boring engineering work.
What is the right data format to move around? JSON?
It's just a saying. All we do is move protos from one service to another.
JSON is definitely not the right stuff.
Re: Zanzibar: Consistent, Global Authorization System
#27Am I alone in thinking that 99.999% measured availability for a service so completely in the critical path for almost everything is relatively low? Phrased another way, when it is not availability, do end users experience service disruption, and if not, how is that mitigated?
Overall uptime isn't the only stat that matters here, the distribution of downtime matters too. One 15 minute outage in three years is a lot worse than 900 1 second outages over that same time period. One second blips are a part of the web, we click refresh and move on--not even knowing who's fault it was.
Re: Zanzibar: Consistent, Global Authorization System
#28Am I alone in thinking that 99.999% measured availability for a service so completely in the critical path for almost everything is relatively low? Phrased another way, when it is not availability, do end users experience service disruption, and if not, how is that mitigated?
It kinda depends what availability means? That .001% unavailability might be degraded service, might be .001% of clients having a bad time across the entire year, might be 'acts of god' (i.e. broken CPUs and the like). This kind of service is also usually fairly low down on the stack, and higher level applications can usually degrade gracefully. If they couldn't, complex applications such as Google would fail to operate; there's always _something_ broken.
Re: Zanzibar: Consistent, Global Authorization System
#29Am I alone in thinking that 99.999% measured availability for a service so completely in the critical path for almost everything is relatively low? Phrased another way, when it is not availability, do end users experience service disruption, and if not, how is that mitigated?
It says greater than 5 nines, and it's usually much greater - in usual times, these core services are usually at six or seven nines as measured client side. But it doesn't take long at three nines to destroy your five nine SLA.
The other portion is client side retry logic. It's incredibly easy for developers to mark a lookup with a retry policy and timer, and one of the reasons that that latency is so low is so that even if there's a timeout, the pageview can succeed. The application code doesn't see the error at all if the retry is successful, it just takes longer. The retry code is very good and it's already known at the first rpc call where the retry should go - the connection pool maintains connections to multiple independent servers.
Re: Zanzibar: Consistent, Global Authorization System
#30Earlier quoted context omitted.
The harsh truth of working at Google is that in the end you are moving protobufs from one place to another. They have the most talented people in the world but those people still have to do some boring engineering work.
But you can reduce any job to this can't you? Pretty much all engineering is just moving some strings around.
Work in finance and you can move integers around!