Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Binary Authorization for Borg

cloud.google.com

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Binary Authorization for Borg

#31

Post by ithkuil »

mayakacz wrote: post #29
packetslave wrote: post #20

Earlier quoted context omitted.

That's a bit of disingenuous reply... Binary Authorization for Borg is for verifying binaries running inside Google, not code installed on end-user machines. Having the authorization be "publicly verifiable" makes no sense.

Agreed with this statement. It's a best practice generally to verify all software updates originate from a particular source before applying them in your environment. Most over the wire updates do this. What's different with Binary Authorization for Borg is that within Google, that last verification step means more than just "came from Google", but "came from Google and went through all previous necessary checks", be…

in-toto.io also addresses the "proof it went through some steps". How would you compare the two systems?

Re: Binary Authorization for Borg

#33

Post by mayakacz »

GuyPostington wrote: post #13
seriesf wrote: post #12

One thing that really squicked me out when I left Google is how other companies, even large and sophisticated ones, are using all kinds of garbage that comes from canonical or red hat or percona, and they have NO IDEA what's in there. Say what you want about google's NIH culture, but in regards to code provenance and verifiable builds they are doing the right thing and many others are not.

Can you give an example of this garbage?

I think 'garbage' is a strong word, but I believe what the original poster is trying to say is that there are a lot of binaries, packages, and libraries that most organizations will consume from upstream, and not verify directly. This requires either trust on a third party (often many third parties - in the case of open source), or more intense validation of those components and any changes to those components.

Binary Authorization for Borg performs verification for pieces that come out of Google's CI/CD pipeline. For third party code, see in the doc, "When importing changes from third party or open source code, we verify that the change is appropriate (for example, the latest version)."

Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.

Re: Binary Authorization for Borg

#34

Post by mayakacz »

skybrian wrote: post #19
shadowgovt wrote: post #9

Earlier quoted context omitted.

It's more a defense against getting NSA'd (via the specific threat model of an attacker secretly replacing a security service with an implementation that looks very similar but is much easier to crack).

More generally you might say it supports rule of law. If something happens according to procedure then it's ok. You might not think that's much of a guarantee, but it beats the alternative where things happen due to shadow processes.

I think that's right. I would strengthen that statement slightly - it's about ensuring that no actor - whether an insider, or someone who has stolen their credentials, or otherwise compromised them - can perform an action that single handedly accesses user data, without it being known to another actor - via access logs, via approvals, etc.

In terms of the upstream introduction of a new vulnerability, Binary Authorization for Borg can only verify that the code was in fact merged. See the section on third party code, "When importing changes from third party or open source code, we verify that the change is appropriate (for example, the latest version)."

Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.

Re: Binary Authorization for Borg

#35

Post by tgma »

kevincox wrote: post #23
rsync wrote: post #4

In case anyone else wonders what 'borg' is: "Our infrastructure is containerized, using a cluster management system called Borg." I was hoping they had some predictable, indexed build for borg backup[1]. [1] https://www.stavros.io/posts/holy-grail-backups/

Borg is basically the internal predecessor to Kubernetes.

Predecessor may imply "worse" or "outdated" (although may not be the intent of the OP). I want to clarify that is definitely not the case: Kubernetes is a joke compared to Borg when it comes to running Google workloads (on many dimensions, most importantly scale).

Re: Binary Authorization for Borg

#36

Post by mayakacz »

ithkuil wrote: post #31
mayakacz wrote: post #29

Earlier quoted context omitted.

Agreed with this statement. It's a best practice generally to verify all software updates originate from a particular source before applying them in your environment. Most over the wire updates do this. What's different with Binary Authorization for Borg is that within Google, that last verification step means more than just "came from Google", but "came from Google and went through all previous necessary checks", be…

in-toto.io also addresses the "proof it went through some steps". How would you compare the two systems?

The biggest difference for me is that in-toto allows you to define any set of upstream metadata requirements, in a very open format, and Binary Authorization has a set of centrally defined requirements, that teams tend to implement in tranches, to meet minimum requirements. It may sound better to have a freeform format, but in practice, I've found that it makes it harder for people to know what they should actually do. In Binary Authorization for Borg, services still define service-specific policies, but pick from a previously defined set of potential requirements. See the section on service-specific policies: https://cloud.google.com/security/binary-authorization-for-b...

You can more easily compare Grafeas and Kritis (OSS projects Google developed, which are similar to GCR Vulnerability Scanning and Binary Authorization for GKE), to in-toto. In fact, I gave a talk covering some of the options for this here: https://youtu.be/uDWXKKEO8NU?t=1314

Disclosure: I work at Google and helped write this whitepaper on Binary Authorization for Borg.

Re: Binary Authorization for Borg

#37

Post by kevincox »

tgma wrote: post #35
kevincox wrote: post #23

Earlier quoted context omitted.

Borg is basically the internal predecessor to Kubernetes.

Predecessor may imply "worse" or "outdated" (although may not be the intent of the OP). I want to clarify that is definitely not the case: Kubernetes is a joke compared to Borg when it comes to running Google workloads (on many dimensions, most importantly scale).

I don't know about joke. Kubernetes has a much better design and interface than Borg. Often times I am frustrated trying to work around design flaws in Borg.

However yes, Borg has been enriched with about a decade of features and fixes that Google needs. A replacement of Borg is many, many years away.

Re: Binary Authorization for Borg

#38

Post by brown9-2 »

jsty wrote: post #15
seriesf wrote: post #12

One thing that really squicked me out when I left Google is how other companies, even large and sophisticated ones, are using all kinds of garbage that comes from canonical or red hat or percona, and they have NO IDEA what's in there. Say what you want about google's NIH culture, but in regards to code provenance and verifiable builds they are doing the right thing and many others are not.

Whilst it would be nice if everyone had the time and resources to code review and build their entire source dependency tree, is this ever going to be a reality for the long tail of enterprises who struggle with even resourcing / recruiting for their current workload? I think the vast majority are going to continue outsourcing this responsibility onto enterprise distros / vendors for a long time to come.

I think things would be easier for the long tail with more investment from all in the tools space - better support for monolithic repos, unified CI/CD systems, etc.

Re: Binary Authorization for Borg

#39

Post by Thaxll »

tgma wrote: post #35
kevincox wrote: post #23

Earlier quoted context omitted.

Borg is basically the internal predecessor to Kubernetes.

Predecessor may imply "worse" or "outdated" (although may not be the intent of the OP). I want to clarify that is definitely not the case: Kubernetes is a joke compared to Borg when it comes to running Google workloads (on many dimensions, most importantly scale).

It's less and less relevant, a lot of Google things can run on the current limits for k8s: 5000 nodes / 150k pods in a single cluster.

Not everything at Google is large, many other compagnies run very large infra like Google scale on Kubernetes using multiple clusters with federation / regions ect ...

Re: Binary Authorization for Borg

#40

Post by philsnow »

> Adopting similar controls in your organization

> Figure out how to manage third party code.

> Many of the CI/CD controls we describe in this paper are placed where your code is developed, reviewed, and maintained by one organization. If you are in this situation, consider how you will include third party code as part of your policy requirements. For example, you could initially exempt the code, while you move towards an ideal state of keeping a repository of all third party code used, and regularly vet that code against your security requirements.

I don't know how much third party code is in use at Google these days, but I'd be curious to know if there is a formal effort at cataloging most-often used / most-sensitive third party code and prioritizing reviews of it.

I've thought about the problem of vetting programming language packages (pypi, npm, rubygems, whatever) off and on. It seems like the only two tenable strategies are "don't pin anything / always use tip of master" and "freeze deps, vet transitive deps at that frozen point, vendor the corresponding deps, and if you ever need to update a requirement for a feature or bugfix, go through the process again".

The latter seems like it could be out-sourced to a certain degree, if you trusted other organizations to "vet transitive deps".

Post reply on HN