Live data from Hacker News

Binary Authorization for Borg

cloud.google.com

51–60 of 60 posts

Re: Binary Authorization for Borg

#51
post #44

At Microsoft, we just require all binaries to be signed on production systems. Some systems are configured to block execution 9f unsigned code. Where we can't do that, monitoring cuts an immediate sev-2 and wakes us up if any unsigned code is executed. Does Linux not have a way to run only signed ELFs?

The second time I read your comment, I read it a little differently, and I the second part of your response sounded AMAZING. monitoring cuts an immediate sev-2 [on systems that cannot be configured to block execution of unsigned code] if any unsigned code is executed I can't wrap my brain around how it would be possible to detect " any unsigned code is executed" on a system that can't ensure the kernel it booted from…

I think anyone can set it up, it's a feature you can administer through group policy. Actually the entire chain of boot from the firmware to the kernel and all userspace is verified. EFI is verified by Secure Boot (through the TPM), which only runs signed kernels, which only runs signed drivers and signed kernel32/user32/etc. So during a normal boot, all running code has been signed and verified.

If that group policy setting I mentioned is enabled, then running unsigned code will either (depending on the setting) write a security event to the system log, which is uploaded and causes a sev-2, or (if set to strict) simply blocked, with a "Windows has protected your computer" message.

It's definitely available publicly - I've personally imaged servers with stock Windows Server 2019 and slipstreamed our internal monitoring agent (which doesn't do anything magic, it just watched the Windows Security ETW log), and personally gotten sev-2s when I tried running unsigned code.

Hence I really wonder why Linux doesn't do this.. I thought SELinux or something might; it could make things slower but my system runs quite fast so I suspect they keep a volatile cache of verified SHAs.

(Also I should mention that even assembly references are signed, so unsigned assemblies refuse to load.)

Re: Binary Authorization for Borg

#52
post #50

At Microsoft, we just require all binaries to be signed on production systems. Some systems are configured to block execution 9f unsigned code. Where we can't do that, monitoring cuts an immediate sev-2 and wakes us up if any unsigned code is executed. Does Linux not have a way to run only signed ELFs?

If a single person can generate a signed binary by themselves, then restricting execution to signed binaries does not address the threat model in the article.

You can't. Only official builds are signed, and two different people have to submit and approve PRs.

Re: Binary Authorization for Borg

#53
post #48
post #47

Earlier quoted context omitted.

AFAIK, it wouldn't be able to unless Python or Ruby provide a way to verify the signatures of the code they're executing, but even then ... that's not really what code signing can do/is for. Start with Intellicode/windows SmartScreen signing -- I have a code signing certificate - cost $100, had to bring a few forms of identification to a notary public, prove a few things about my physical address (which was awesome w…

Re (1), when you say "don't allow programs that can execute arbitrary, unsigned/unverified code" -- does this mean we are blocking all scripting languages? For (2) and (3), I agree that capabilities and containers are very important, but they are not really related to code-signing -- either python is signed or not. So I don't see how code signing + python can co-exist? Once you allowed your python (ruby, perl) binary…

PowerShell is set to a restricted language mode for example after enabling enforced UMCI, and even .VBS files can be signed.

(the signature appears as a comment block in the file in that case, and is then checked by the system for Windows built-in scripting languages)

Re: Binary Authorization for Borg

#54

Earlier quoted context omitted.

That's a bit of disingenuous reply... Binary Authorization for Borg is for verifying binaries running inside Google, not code installed on end-user machines. Having the authorization be "publicly verifiable" makes no sense.

Agreed with this statement. It's a best practice generally to verify all software updates originate from a particular source before applying them in your environment. Most over the wire updates do this. What's different with Binary Authorization for Borg is that within Google, that last verification step means more than just "came from Google", but "came from Google and went through all previous necessary checks", be…

I am perfectly aware that Binary Authorization for Borg is for binaries running inside Google.

I am saying that our solution provides almost the opposite: publicly-verifiable assurance that you are running legitimate binaries, despite being built by automation

Re: Binary Authorization for Borg

#55
post #18

On a related note, we have built an E2E-verified, tamper-evident CI/CD pipeline for the Datadog Agent integrations [1]: the Agent will trust and install only integrations that correspond to source code that have signed by our developers. If there is an attack anywhere between our developers and end-users, it will be caught. Unlike Binary Authorization for Borg, our security guarantees are publicly verifiable. [1] htt…

I saw this before and meant to post about it, because it's really neat.

Thanks, Thomas!

Re: Binary Authorization for Borg

#56
post #50

Earlier quoted context omitted.

If a single person can generate a signed binary by themselves, then restricting execution to signed binaries does not address the threat model in the article.

You can't. Only official builds are signed, and two different people have to submit and approve PRs.

That's a big difference from the starting state that Google had, which was that a single person could create a signed production binary from unsubmitted code all by themselves.

(It was very convenient for iterating on one-off fixes in production in an emergency, but you would rightly question how someone gets into that position in the first place. Plus there was no guarantee that the code would ever get submitted, and post-fix code review might cause the code to be subtly broken prior to being committed to the monorepo.)

Re: Binary Authorization for Borg

#57

Earlier quoted context omitted.

I know this is pedantic, but I'd argue is an inspiration for, in its present state kubernetes is unable to scale to a datacenter, let alone globally at google scale.

I will agree that it was the inspiration. And I think there is slow movement towards scaling kubernetes. Note that Borg isn't a global services. There are many instances in different locations. Borg also doesn't scale globally.

Isn't Borg usually scaled to few rows of racks? i.e. not even spanning whole building, yet alone datacenter.

Re: Binary Authorization for Borg

#58
post #44

Earlier quoted context omitted.

The second time I read your comment, I read it a little differently, and I the second part of your response sounded AMAZING. monitoring cuts an immediate sev-2 [on systems that cannot be configured to block execution of unsigned code] if any unsigned code is executed I can't wrap my brain around how it would be possible to detect " any unsigned code is executed" on a system that can't ensure the kernel it booted from…

I think anyone can set it up, it's a feature you can administer through group policy. Actually the entire chain of boot from the firmware to the kernel and all userspace is verified. EFI is verified by Secure Boot (through the TPM), which only runs signed kernels, which only runs signed drivers and signed kernel32/user32/etc. So during a normal boot, all running code has been signed and verified. If that group policy…

There's IMA subsystem that generally does the same, providing verification for executables at least ones that are linked and loaded normally - for obvious reasons it can't exactly fix cases where you have a signed binary that loads unsigned code into itself.

The system supports integration with TPM-stored keys, so you can bind to kernel verification or the whole firmware chain of trust (a question of which PCRs you end up using to bind the keys).

Unfortunately, it's not well known (especially if you don't compile your own kernel reading through all the options), and generally, not many distributions look into providing it - plus everything involving SecureBoot and TPM has to deal with poisoned opinions in open source community.

Re: Binary Authorization for Borg

#60
post #44

Earlier quoted context omitted.

The second time I read your comment, I read it a little differently, and I the second part of your response sounded AMAZING. monitoring cuts an immediate sev-2 [on systems that cannot be configured to block execution of unsigned code] if any unsigned code is executed I can't wrap my brain around how it would be possible to detect " any unsigned code is executed" on a system that can't ensure the kernel it booted from…

I think anyone can set it up, it's a feature you can administer through group policy. Actually the entire chain of boot from the firmware to the kernel and all userspace is verified. EFI is verified by Secure Boot (through the TPM), which only runs signed kernels, which only runs signed drivers and signed kernel32/user32/etc. So during a normal boot, all running code has been signed and verified. If that group policy…

Thanks for the reply -- I guess what I'm wondering is how can reliably alert about execution of unsigned code on a system that cannot prevent the kernel/OS/firmware from being replaced by something evil. Based on your reply, it sounds like that's a pre-requisite[0], otherwise evil-kernel could simply report that signatures are passing for everything under its control.

I can think of several (imperfect) approaches for protecting a system that meets those per-requisites but is incapable of, say, validating userland programs -- though even that would be tricky to do reliably unless the kernel is checking signatures on execution, rather than passively (via filesystem checks, etc). This is more a fault of any system that "alerts when something out-of-policy happens" vs "prevents something out-of-policy from happening"

The part I have a difficult wrapping my brain around is what kind of system exists with those limitations? If the system is capable of verifying that it is running a signed firmware, bootloader and kernel (along with any other components that are doing signature verification), why isn't that same system capable of preventing every unsigned application from running, eliminating the need to issue a sev-2 for the impossible situation of "it's running unsigned code unexpectedly". The two major examples that come to mind -- Windows and Linux -- handle both requirements (albiet, with additional software on the Linux side/additional configuration on the Windows side).

Recent Windows versions do not have this limitation, AFAIK. Intellicode/code signing has been included for quite some time. It's mature, a lot of it is enabled by default with balanced default settings and as you mentioned, it's not terribly difficult to set up a system to be very strict with its handling of signed code execution[1]. I should clarify -- despite my last comment -- that I believe the way Microsoft implemented this to be well thought out. Though my Dad[2] will probably always click through the SmartScreen message, no matter how ugly it looks, and it can certainly offer a false sense of security for those who leave it in that default configuration, it still adds a nice layer to the onion (without being obnoxious) and serves as a foundation/required component to enable strict policies[3]. And I prefer this approach to a more strict "walled garden" approach -- freedom/usefulness has a security cost[4].

On the Linux side, depending on the distribution, this may or may not be available "out of the box". I know the distribution I run -- openSUSE Tumbleweed -- enforcing bootloader/kernel signing and enabling trusted boot is straight forward (simple if your hardware is natively handled by everything out-of-the-box, but still relatively easy if you have to compile in a module, AFAIK -- having never gone through the pain, myself). And there are a few options after that for userland that are comprehensive in that the aim isn't just to cover elf executables but to ensure that the important parts of the system are cryptographically verified to be unaltered[5].

[0] Assuming the kernel handles signature verification, or minimally verifies the signature of whatever does handle that.

[1] PowerShell is a great example; though it frustrates the hell out of people (me, and anecdotally, everyone I've worked with immediately after trying to run their first script on a new OS load), I love that the default configuration is (maybe was?) "Signed Only", requiring a bit of work (far less in the last version of Windows 10 that I used) to allow unsigned script execution, paired with a permissions system that lets administrators prevent lower security accounts from altering that setting (which also defaults/might only be allowed to be set by administrators in the first place).

[2] Not picking on him -- he was a heck of a power user back in the day, but thinking the average Joe/Jane who re-uses a handful of passwords (just got the parents out of this habit) and are otherwise oblivious to security warnings.

[3] In some ways, it's an extension of the GPO executable white-list -- which was easily bypassed by renaming evil-executable.exe to something that is allowed. Now, contents are checked and the white-lists are tiered (i.e. video/sound drivers must be signed by the user and an MS certificate, which doesn't happen until the program is evaluated ... IIRC, of course).

[4] A computer that's melted down into a mass of plastic/metal hack-proof,... and use-proof. Less snarky -- a computer that has no hardware capable of allowing it to communicate on a network is going to be many orders of magnitude more difficult to breach by simply following good practices on the physical-security front, but it'll be similarly useless if that system was intended to host a publicly accessible web site.

[5] As was mentioned, before, even saying "ensure all executable code is signed" isn't enough for some environments -- ensuring configuration is unaltered is an obvious one. Even signing state data is necessary in some cases, i.e. a system is hibernated, drive is removed/file is altered, and the system is booted, loading evil-hiberfile.sys. Sheesh, security work is fun -- close one hole and the "what abouts" start popping up everywhere else.

Post reply on HN