Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zanzibar: Consistent, Global Authorization System

ai.google

Post reply on HN
Print view View original

81–90 of 137 posts

Re: Zanzibar: Consistent, Global Authorization System

#81

Post by manigandham »

idlewords wrote: post #38
colesantiago wrote: post #6

I love reading about Google's systems, but I wish I could work on those problems at scale, that is my dream really. I wonder what more systems Google has that we don't know about. I know Borg has become what we know as k8s but surely there must be more things that Google has made internally that are not open source. Curious about this and would like to know more about it from anyone in the trenches at Google.

Wait until you get a glimpse into the exciting world of real-time ad bidding. It's every engineer's dream!

Adtech is interesting because of the scale, complexity, and timing required compared to many other software projects. It gets a bad look but the engineering involved is not boring.

Re: Zanzibar: Consistent, Global Authorization System

#82

Post by gen220 »

victor106 wrote: post #37

What do other large (non-google scale) to medium companies use for authorization? Can anyone recommend open source (preferably) or close source products?

We use LDAP for managing group memberships (i.e. person x is a member of `engineering` and `eng_team_y`; only members of `eng_team_y` can change the deployment status of service Z). We then define ACLs for these groups. IDK how they are enforced, but they're visible/malleable via Ansible recipes, such that the process of adding permissions for your group (or user) involves submitting a diff to said Ansible recipe and getting approval from an SRE.

In practice, we use Kerberos to obtain/distribute authorization tokens, which live for less than 24 hours. The authorization-value of these tokens is determined by the LDAP affinities of the bearer. If everything is configured correctly (which it always is, until you need new permissions / switch teams), all you have to do is auth with kerberos at the beginning of each day. We have ~200 engineers.

Re: Zanzibar: Consistent, Global Authorization System

#83

Post by Iv »

koboll wrote: post #50
wallflower wrote: post #20

> There's also a story behind that project name. That is not the original project name. The original project name was force-removed by my SVP. Once my hands are free again, I can explain https://mobile.twitter.com/LeaKissner/status/113663143751427...

Yeah, that's kinda understandable. Although I'm not sure how the current name fits the project any better.

It sounds pretty appropriate. This is another SF reference "Stand on Zanzibar", one of the earliest inspirations for cyberpunk, in which we are routinely remembered that all of humanity would be able to stand on the small island of Zanzibar, but that growth is problematic.

https://en.wikipedia.org/wiki/Stand_on_Zanzibar

Re: Zanzibar: Consistent, Global Authorization System

#84

Post by manigandham »

xfitm3 wrote: post #77
shereadsthenews wrote: post #43

Earlier quoted context omitted.

The competitive nature of Googlers is what make Google a very fun place to work.

Run in the rat race for a decade or two and report back.

The rat race typically describes the cycle of working to live and living to work without much else going on.

Competition at work is something different, and you're always in competition with other companies and people to survive anyway.

Re: Zanzibar: Consistent, Global Authorization System

#85

Post by manigandham »

Boulth wrote: post #7
colesantiago wrote: post #6

I love reading about Google's systems, but I wish I could work on those problems at scale, that is my dream really. I wonder what more systems Google has that we don't know about. I know Borg has become what we know as k8s but surely there must be more things that Google has made internally that are not open source. Curious about this and would like to know more about it from anyone in the trenches at Google.

On the other hand Google engineers are vendor-locked-in using Google specific tech one way or another.

Why? The build-or-buy trade-off is very different at Google scale and this is one of the few organizations that can build everything in-house for their specific needs.

Re: Zanzibar: Consistent, Global Authorization System

#86

Post by kjeetgill »

Iv wrote: post #83
koboll wrote: post #50

Earlier quoted context omitted.

Yeah, that's kinda understandable. Although I'm not sure how the current name fits the project any better.

It sounds pretty appropriate. This is another SF reference "Stand on Zanzibar", one of the earliest inspirations for cyberpunk, in which we are routinely remembered that all of humanity would be able to stand on the small island of Zanzibar, but that growth is problematic. https://en.wikipedia.org/wiki/Stand_on_Zanzibar

For others that were confused as I was: SF here is Science Fiction not San Francisco.

Re: Zanzibar: Consistent, Global Authorization System

#87

Post by kjeetgill »

sametmax wrote: post #58
shereadsthenews wrote: post #40

Earlier quoted context omitted.

Yeah the whole time I was there every time I had to use bigtable or Spanner I always muttered to myself “I wish I could be using some free software garbage instead of this proprietary Google stuff right now.” Every Googler secretly yearns for the performance, reliability, and elegance of MySQL.

Uncool, given that Google only exist thanks to free software. Google started as a C, Java and Python shops. Android is born out of a Linux Kernel. If I remember well, long ago, the initial database they used was actually an in-shop fork of MySQL. Easy to say bad things about free software now that you have billions and hundred of geniuses to work on your own stuff. Actually, this kind of comment reminds me of the "li…

I agree with you both! I don't think Google (or probably even the parent) meant to disrespect or discredit the value and contributions of open source. But internal, non-open platforms aren't really "vendor lock-in" or just NIH. They're often much of much higher quality (if you have the resources) simply by solving the exact problems you have directly.

Disclaimer: I've been bitter about MySQL-for-everything-ism lately too but think Java is pure heaven.

Re: Zanzibar: Consistent, Global Authorization System

#88

Post by the-rc »

jamesblonde wrote: post #64
colesantiago wrote: post #6

I love reading about Google's systems, but I wish I could work on those problems at scale, that is my dream really. I wonder what more systems Google has that we don't know about. I know Borg has become what we know as k8s but surely there must be more things that Google has made internally that are not open source. Curious about this and would like to know more about it from anyone in the trenches at Google.

Colossus is their data center scale filesystem. They dont talk about it..

Colossus is actually the only project I can think of for which they had one of the leaders sit down with Kirk McKusick and have a chat for ACM Queue, instead of a paper. https://queue.acm.org/detail.cfm?id=1594206

Re: Zanzibar: Consistent, Global Authorization System

#89

Post by jacques_chester »

stingraycharles wrote: post #18

Am I alone in thinking that 99.999% measured availability for a service so completely in the critical path for almost everything is relatively low? Phrased another way, when it is not availability, do end users experience service disruption, and if not, how is that mitigated?

You may have missed how they're defining it:

> We define availability as the fraction of “qualified” RPCs the service answers successfully within latency thresholds: 5 seconds for a Safe request, and 15 seconds for a Recent request as leader re-election in Spanner may take up to 10 seconds. ... To compute availability, we aggregate success ratios over 90-day windows averaged across clusters. Figure 5 shows Zanzibar’s availability as measured by these probers. Availability has remained above 99.999% over the past 3 years ofoperation at Google. In other words, for every quarter, Zanzibar has less than 2 minutes of global downtime and fewer than 13 minutes when the global error ratio exceeds 10%.

Basically, they're counting by number of requests. That's fairly typical for Google, who in their SRE book point out that measuring only total outages is a poor indicator of actual user experiences. Imagine if you had an electric company that had frequent brownouts and rolling blackouts but bragged about never having a total blackout. You'd be fairly unimpressed.

Google SREs also make the point that beyond five nines, your efforts are rendered moot by reliability issues you cannot control. Mostly network issues. If you have 99.99999% reliability but the mobile data network only has 99.99%, you've wasted a lot of money on something most folks will never notice.

Re: Zanzibar: Consistent, Global Authorization System

#90

Post by atombender »

anonygler wrote: post #71
colesantiago wrote: post #6

I love reading about Google's systems, but I wish I could work on those problems at scale, that is my dream really. I wonder what more systems Google has that we don't know about. I know Borg has become what we know as k8s but surely there must be more things that Google has made internally that are not open source. Curious about this and would like to know more about it from anyone in the trenches at Google.

The most impressive part about Google is how its emphasis on internal standards has allowed it to build some really impressive stuff. Eg, You can do a sql join on any dataset, in any datacenter. You can turn any query into a hosted visualization. Every test invocation is streamed to a central server and results can be shared with a url. There’s more, but those are my two favorites.

Is the join a Spanner query, or is there a system on top of a Spanner that federates/aggregates databases?
Post reply on HN