Live data from Hacker News

Binary Authorization for Borg

cloud.google.com

41–50 of 60 posts

Re: Binary Authorization for Borg

#41

At Microsoft, we just require all binaries to be signed on production systems. Some systems are configured to block execution 9f unsigned code. Where we can't do that, monitoring cuts an immediate sev-2 and wakes us up if any unsigned code is executed. Does Linux not have a way to run only signed ELFs?

Thanks for that -- I love MS security practices/stories; interesting.

Yes, Linux does have ELF signing. I'm guessing you are speaking more generally about ensuring that "only signed code is allowed to execute", rather than just "making sure ELF binaries are signed" based on the remaining context of your comment.

Similar to Windows, making sure that exe files are signed isn't enough (PowerShell, drivers, kernel, firmware, etc) -- there's PowerShell scripts, etc, and "block execution of unsigned code" or even "block privileged execution of unsigned code".

Assuming that "signed code execution", as I frequently discover when I go looking for "how to do Foo Linux", there's more than one[0] way, depending on what the need/device/system is. Windows is in a lot of places you don't expect -- ATMs, IoT devices, etc -- Linux is ... it'd be easier to come up with a list of device types that haven't had a Linux kernel running on them. LWN had a write up in 2017 -- I know I've read more current, but theirs was a good summary and answers your question. The Linux Integrity Measurement Architecture (IMA)[2] is a more complete approach. Those are the more "general-use options" that I was aware of.

[0] Often at least 8; and there's usually a few of them arguing with eachother over something that's between the extremes of "who's dad would actually win in a boxing match" and ... religion. /s

[1] https://lwn.net/Articles/733431/

[2] Wiki Link- https://sourceforge.net/p/linux-ima/wiki/Home/ Good write-up https://lwn.net/Articles/488906/

Re: Binary Authorization for Borg

#42

At Microsoft, we just require all binaries to be signed on production systems. Some systems are configured to block execution 9f unsigned code. Where we can't do that, monitoring cuts an immediate sev-2 and wakes us up if any unsigned code is executed. Does Linux not have a way to run only signed ELFs?

How does this work with scripts? Python or Ruby script can do a lot of damage.

Or does MS prohibit all scripting languages on production systems?

Re: Binary Authorization for Borg

#43
post #35

Earlier quoted context omitted.

Borg is basically the internal predecessor to Kubernetes.

Predecessor may imply "worse" or "outdated" (although may not be the intent of the OP). I want to clarify that is definitely not the case: Kubernetes is a joke compared to Borg when it comes to running Google workloads (on many dimensions, most importantly scale).

This post highlights why many start disliking Google - its that vibe of superiority that comes across often. These days Google workload (e.g. qps) isn't that unique anymore...

Moving Borg to Kunernetes will eventually happen because it doesnt make sense to help maintain two systems that solve the same problem. And because Kubernetes is open source it will eventually be superior, because of diverse group of contributions.

Re: Binary Authorization for Borg

#44

At Microsoft, we just require all binaries to be signed on production systems. Some systems are configured to block execution 9f unsigned code. Where we can't do that, monitoring cuts an immediate sev-2 and wakes us up if any unsigned code is executed. Does Linux not have a way to run only signed ELFs?

The second time I read your comment, I read it a little differently, and I the second part of your response sounded AMAZING.

monitoring cuts an immediate sev-2 [on systems that cannot be configured to block execution of unsigned code] if any unsigned code is executed

I can't wrap my brain around how it would be possible to detect "any unsigned code is executed" on a system that can't ensure the kernel it booted from passed a signature check, so I'm guessing that was simplified to avoid a miles-long comment.

I'm not trying to be pedantic -- I'm actually curious if you have any more you can share/point me at -- measures that Microsoft has found effective[0] at reducing the attack surface of those unfortunate devices at the "alert if what's running looks right, but fails a cryptographic check". I mean, GPO has allowed a white-list of executables for as long as I can remember, and there have always been permissions in the OS to control rights "provided everyone is playing by the rules", but the GPO was trivial to defeat[1]. Are there any other interesting measures you can share? Also -- when you talk about "binaries signed on production systems" -- are you referring to Intelli-code capabilities built into Windows[2] or something further?

[0] I'm sure, even they, are guilty of having some tooling that's little more than theatre and wasted CPU cycles in places.

[1] And I'm guessing these are made up of older systems with a larger percentage of software that can be easily fooled into calling into a DLL that it didn't think it was calling into.

[2] I haven't spent a lot of time reading on the topic, but I have a code-signing certificate -- I don't know a lot about the topic, though, was just curious if it's something "I can play with", or if it includes something I can't get (whether or not that's "because it's only available internally", or "it's enterprise expensive"). :)

Re: Binary Authorization for Borg

#45

I led the portion of this project on Borg itself. Security team did most of the security infrastructure, and coordination among almost every large infrastructure system team inside TI. I'll be waiting for them to answer any questions. :)

A comment, not a question: Though I think it was worth the cost, I'd say this was one of the most painful mandates/rollouts I've had to endure. The cost to developer productivity was pretty significant. I would have liked to have seen that impact discussed.

Re: Binary Authorization for Borg

#46
post #43
post #35

Earlier quoted context omitted.

Predecessor may imply "worse" or "outdated" (although may not be the intent of the OP). I want to clarify that is definitely not the case: Kubernetes is a joke compared to Borg when it comes to running Google workloads (on many dimensions, most importantly scale).

This post highlights why many start disliking Google - its that vibe of superiority that comes across often. These days Google workload (e.g. qps) isn't that unique anymore... Moving Borg to Kunernetes will eventually happen because it doesnt make sense to help maintain two systems that solve the same problem. And because Kubernetes is open source it will eventually be superior, because of diverse group of contributi…

Huh? How is superiority relevant here? I don’t work at Google (I have in the past) and both systems come out of that company with some of the same people have worked on both. If anything, Google marketing department would probably prefer for people to believe that Kubernetes ~= Borg to help sell GKE, not the other way around. Kubernetes is basically limited to several thousand hosts. That doesn’t even register at Google scale. Other folks do have high QPS systems but none really use Kubernetes to manage the entire cluster; Facebook comes to mind, for example, with an in house system. I would bet against that prediction; I don’t believe such a thing is even on the roadmap internally.

Re: Binary Authorization for Borg

#47
post #42

At Microsoft, we just require all binaries to be signed on production systems. Some systems are configured to block execution 9f unsigned code. Where we can't do that, monitoring cuts an immediate sev-2 and wakes us up if any unsigned code is executed. Does Linux not have a way to run only signed ELFs?

How does this work with scripts? Python or Ruby script can do a lot of damage. Or does MS prohibit all scripting languages on production systems?

AFAIK, it wouldn't be able to unless Python or Ruby provide a way to verify the signatures of the code they're executing, but even then ... that's not really what code signing can do/is for.

Start with Intellicode/windows SmartScreen signing -- I have a code signing certificate - cost $100, had to bring a few forms of identification to a notary public, prove a few things about my physical address (which was awesome when you use a MVNO and don't have a land-line). At the end of the day, I can sign code that ticks the happy boxes in Windows 10, Visual Studio, etc. Obviously, the certificate signs evil as well as good. And my signature only really says "signed with the certificate that this person purchased".

On the more extreme end, imagine a hypothetical system -- from bootloader to OS and its configuration, everything must be signed with both my key and the key of my buddy Greg. We did everything perfectly so as to ensure that the bits inside these 5 programs are the only bits that have ever been signed with these keys and any other edge cases are covered -- i.e. it's a system that perfectly covers "This computer will do whatever these 5 programs tell it to do", and "We know these 5 programs are guaranteed to be unaltered"

That, alone, isn't a protection against "damage". More than code-signing is necessary for that:

1. Understand what the whatever is that the "allowed" programs do -- and (optionally/extreme cases) don't allow programs that can execute arbitrary, unsigned/unverified code. 2. Use every tool the operating system makes available to enforce what the program is allowed to do. If `python` is needed to run a script that copies file A on a remote server to file B on the local server, a start is to run with permissions that prevents python from damanging more than file A if the intended script were replaced with something-evil.py. How narrow you'll be able to define these rules depends on the capabilities of the operating system/hardware platform. 3. Further isolate (a.l.a. containers/container-like) -- pop it in a sub-environment (container/virtual machine) running a signed image that contains the python and the script -- no idea if docker provides anything here, but there's nothing to stop one from moving the whole thing to a small device with a signed bootloader/image containing the script that refuses to boot if any bit is changed on the device. It just depends on insane you want to get.

It's these permissions systems that prevent "damage" -- really, if the permissions systems were "perfect" they'd let you specify, precisely "in what context the application is allowed to perform what action". Code signing is, arguably, one of these permissions systems, too. Granted, without effective code-signing involved, the permissions checks can be bypassed in an undetectable manner.

I can't speak for MS -- not my employer -- but I worked for a large global telecom doing a mess of infrastructure development centered around security (among others). At the time, code signing/secure boot as it exists today ... I don't even know that it existed. We had a system that was allowed to run a single data translation program. It was possible to make this thing meet "spirit" of the above with a mix of hardware/physical security[0]. We had machines that enforced policies about what was allowed and disallowed. Every services security needs are different, and while there was certainly a "baseline -- everything at least does this", once you started talking wanting the assurances that "ensuring only signed code runs" involved, usually that landed it in the highest-security configuration for everything else (multiple-ring restricted internal white-list-only networks where inbound/outbound is tightly controlled and documented) followed by an analysis of the service to see how little we can give it and still make it work.

[0] I recall a box that ran a single DOS executable at boot from removable media where the drive had been jumpered to be read-only. It was locked in a cabinet in a data-center that myself and 5-or-so others had access to and its purpose was to receive data from a serial connection do dance around with it a bit and upload the resulting file at 1200 baud to God only knows where.

Re: Binary Authorization for Borg

#48
post #47
post #42

Earlier quoted context omitted.

How does this work with scripts? Python or Ruby script can do a lot of damage. Or does MS prohibit all scripting languages on production systems?

AFAIK, it wouldn't be able to unless Python or Ruby provide a way to verify the signatures of the code they're executing, but even then ... that's not really what code signing can do/is for. Start with Intellicode/windows SmartScreen signing -- I have a code signing certificate - cost $100, had to bring a few forms of identification to a notary public, prove a few things about my physical address (which was awesome w…

Re (1), when you say "don't allow programs that can execute arbitrary, unsigned/unverified code" -- does this mean we are blocking all scripting languages?

For (2) and (3), I agree that capabilities and containers are very important, but they are not really related to code-signing -- either python is signed or not.

So I don't see how code signing + python can co-exist? Once you allowed your python (ruby, perl) binary, the "binary whitelist" is pretty useless. Seem like other technologies -- like containers, sandboxes, SELinux-like labeling etc -- is the only way to go.

Re: Binary Authorization for Borg

#49

> Adopting similar controls in your organization > Figure out how to manage third party code. > Many of the CI/CD controls we describe in this paper are placed where your code is developed, reviewed, and maintained by one organization. If you are in this situation, consider how you will include third party code as part of your policy requirements. For example, you could initially exempt the code, while you move towar…

For one thing, all third-party code is checked into one top-level folder in the monorepo.

This is publicly documented here:

https://opensource.google/docs/thirdparty/

This means that dependencies on a third-party library can be found simply by looking at deps lines in BUILD rules. That can then inform which projects you want to run (for example) fuzzers on:

https://opensource.googleblog.com/2016/12/announcing-oss-fuz... https://google.github.io/oss-fuzz/

Re: Binary Authorization for Borg

#50

At Microsoft, we just require all binaries to be signed on production systems. Some systems are configured to block execution 9f unsigned code. Where we can't do that, monitoring cuts an immediate sev-2 and wakes us up if any unsigned code is executed. Does Linux not have a way to run only signed ELFs?

If a single person can generate a signed binary by themselves, then restricting execution to signed binaries does not address the threat model in the article.
Post reply on HN