Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Binary Authorization for Borg

cloud.google.com

Post reply on HN
Print view View original

21–30 of 60 posts

Re: Binary Authorization for Borg

#21

Post by prattmic »

agv123 wrote: post #10
justicezyx wrote: post #5

I led the portion of this project on Borg itself. Security team did most of the security infrastructure, and coordination among almost every large infrastructure system team inside TI. I'll be waiting for them to answer any questions. :)

I've seen references to gVisor being used 'in production' for google app engine && cloud run and so forth. Scanning through recent commits && the github repo this is clearly not the case - there are way too many outstanding issues and outright missing support for various things. Is this another project where it was written in a different language or something and then ported out? Can you clarify?

I work on gVisor, I can answer this!

gVisor is not a rewritten version of an internal tool. The code you see really does run in production for App Engine and Cloud Run. While there are some internal modifications to better integrate with internal infrastructure, the vast majority of the code is identical to open source, critically including all of the system call handling, filesystem, and memory management code.

While browsing through our issues will show that we still have plenty to work on, the vast majority of applications work well inside gVisor.

Re: Binary Authorization for Borg

#22

Post by ithkuil »

packetslave wrote: post #20
trishankdatadog wrote: post #7

On a related note, we have built an E2E-verified, tamper-evident CI/CD pipeline for the Datadog Agent integrations [1]: the Agent will trust and install only integrations that correspond to source code that have signed by our developers. If there is an attack anywhere between our developers and end-users, it will be caught. Unlike Binary Authorization for Borg, our security guarantees are publicly verifiable. [1] htt…

That's a bit of disingenuous reply... Binary Authorization for Borg is for verifying binaries running inside Google, not code installed on end-user machines. Having the authorization be "publicly verifiable" makes no sense.

Publicly verifiable is a (stronger) superset of privately verifiable

Re: Binary Authorization for Borg

#23

Post by kevincox »

rsync wrote: post #4

In case anyone else wonders what 'borg' is: "Our infrastructure is containerized, using a cluster management system called Borg." I was hoping they had some predictable, indexed build for borg backup[1]. [1] https://www.stavros.io/posts/holy-grail-backups/

Borg is basically the internal predecessor to Kubernetes.

Re: Binary Authorization for Borg

#24

Post by pjc50 »

seriesf wrote: post #14
GuyPostington wrote: post #13

Earlier quoted context omitted.

Can you give an example of this garbage?

Literally anything that comes from a vendor in a package? Percona server/toolkit? Every binary package in Ubuntu? The Linux kernel as built and distributed by Red Hat?

Yeah, that's not a definition of garbage that's going to be taken seriously.

Re: Binary Authorization for Borg

#25

Post by seriesf »

pjc50 wrote: post #24
seriesf wrote: post #14

Earlier quoted context omitted.

Literally anything that comes from a vendor in a package? Percona server/toolkit? Every binary package in Ubuntu? The Linux kernel as built and distributed by Red Hat?

Yeah, that's not a definition of garbage that's going to be taken seriously.

You're right; Google is the only big tech that takes insider risk seriously.

Re: Binary Authorization for Borg

#26

Post by KaiserPro »

kevincox wrote: post #23
rsync wrote: post #4

In case anyone else wonders what 'borg' is: "Our infrastructure is containerized, using a cluster management system called Borg." I was hoping they had some predictable, indexed build for borg backup[1]. [1] https://www.stavros.io/posts/holy-grail-backups/

Borg is basically the internal predecessor to Kubernetes.

I know this is pedantic, but I'd argue is an inspiration for, in its present state kubernetes is unable to scale to a datacenter, let alone globally at google scale.

Re: Binary Authorization for Borg

#27

Post by KaiserPro »

seriesf wrote: post #12

One thing that really squicked me out when I left Google is how other companies, even large and sophisticated ones, are using all kinds of garbage that comes from canonical or red hat or percona, and they have NO IDEA what's in there. Say what you want about google's NIH culture, but in regards to code provenance and verifiable builds they are doing the right thing and many others are not.

It depends on your threat model.

If you are a large company tech company (1-5k employees) there are far bigger risks than dodgy binary builds from upstream. (like leaked API keys to github...)

However, if you are a hyperscale, high value company (ie a place which has enough data or digital cash to be worth dicking with) then its a worthy problem.

Re: Binary Authorization for Borg

#28

Post by kevincox »

KaiserPro wrote: post #26
kevincox wrote: post #23

Earlier quoted context omitted.

Borg is basically the internal predecessor to Kubernetes.

I know this is pedantic, but I'd argue is an inspiration for, in its present state kubernetes is unable to scale to a datacenter, let alone globally at google scale.

I will agree that it was the inspiration. And I think there is slow movement towards scaling kubernetes.

Note that Borg isn't a global services. There are many instances in different locations. Borg also doesn't scale globally.

Re: Binary Authorization for Borg

#29

Post by mayakacz »

packetslave wrote: post #20
trishankdatadog wrote: post #7

On a related note, we have built an E2E-verified, tamper-evident CI/CD pipeline for the Datadog Agent integrations [1]: the Agent will trust and install only integrations that correspond to source code that have signed by our developers. If there is an attack anywhere between our developers and end-users, it will be caught. Unlike Binary Authorization for Borg, our security guarantees are publicly verifiable. [1] htt…

That's a bit of disingenuous reply... Binary Authorization for Borg is for verifying binaries running inside Google, not code installed on end-user machines. Having the authorization be "publicly verifiable" makes no sense.

Agreed with this statement. It's a best practice generally to verify all software updates originate from a particular source before applying them in your environment. Most over the wire updates do this. What's different with Binary Authorization for Borg is that within Google, that last verification step means more than just "came from Google", but "came from Google and went through all previous necessary checks", because of the way the CI/CD system works together.

Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.

Re: Binary Authorization for Borg

#30

Post by mayakacz »

tptacek wrote: post #2

Has anyone outside of Google implemented something similar in spirit to this for K8s or ECS? What was the threat model you were considering when you built it? Was it worth it?

Yes, there's a few listed in this blog post: https://cloud.google.com/blog/products/identity-security/bey... - Kubernetes admission controllers, OSS part of Kubernetes: https://kubernetes.io/docs/reference/access-authn-authz/admi... - Kritis, OSS: https://opensource.google/projects/kritis - OPA Gatekeeper, OSS: https://github.com/open-policy-agent/gatekeeper - Binary Authorization on GKE/Anthos: https://cloud.google.com/binary-authorization/ They don't all do all the pieces. The hardest part is going to be integrating whatever enforcement solution you choose with your upstream CI/CD pipeline.

Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.

Post reply on HN