Patching is hard; so what?

#1

Post by runesoerensen » Fri, Sep 15, 2017, 10:55 PM UTC

Patching is hard; so what?
blog.cryptographyengineering.com

Re: Patching is hard; so what?

#2

Post by empath75 » Sat, Sep 16, 2017, 12:12 AM UTC

I know equifax is probably dealing with a hairy legacy system, but really you should design all your systems to be able to push out a fully tested blue/green deploy in hours not months. There's no excuse not to with Jenkins/aws/docker/etc.

You should be testing and pushing updates regularly as a matter of course, not as a red alert only when security vulnerabilities are published.

Re: Patching is hard; so what?

#3

Post by chasb » Sat, Sep 16, 2017, 12:15 AM UTC

For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason.

If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely.

That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.

Re: Patching is hard; so what?

#4

Post by lucb1e » Sat, Sep 16, 2017, 12:21 AM UTC

TL;DR: patching is hard, so use defense in depth to minimize/mitigate the impact of individual components becoming insecure from time to time.

Re: Patching is hard; so what?

#5

Post by inetknght » Sat, Sep 16, 2017, 12:31 AM UTC

chasb wrote: post #3

For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason. If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely. That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.

Even if they learn of a CVE there's no guarantee a patch exists for it. You might even be stuck with no alternative but to disable whatever the attack vector is... which could be key to your business.

Re: Patching is hard; so what?

#6

Post by gerdesj » Sat, Sep 16, 2017, 12:46 AM UTC

chasb wrote: post #3

For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason. If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely. That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.

What on earth is a "DevOps model"? (yes I did see your ahem model).

Please forgive me if I don't hire you to keep my kool ... whatever ... webby thing safe. I understand that you are commentating in a second language but you used the DevOps safe word and automatically lose.

Re: Patching is hard; so what?

#7

Post by guelo » Sat, Sep 16, 2017, 1:04 AM UTC

Patching isn't that hard. Once Equifax finally noticed the intrusion (intrusion detection system?), they took the system down and patched it within a day.

Re: Patching is hard; so what?

#8

Post by shawkinaw » Sat, Sep 16, 2017, 2:20 AM UTC

Maybe patching is hard. But the real question is why one vuln in their web framework allowed unfettered access to basically all of their data. There are much deeper problems here than slow patching.

Re: Patching is hard; so what?

#9

Post by rdtsc » Sat, Sep 16, 2017, 2:42 AM UTC

chasb wrote: post #3

For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason. If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely. That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.

Exactly. I'd say the check of how fast can a security patch be deployed from the moment upstream OS or library releases is a good benchmark for a team/system and should be evaluated at the same level as log audits, what ports are open and what openssl versions are running.

> it's just underinvested in early on.

And for cases where there is no personal or financial data involved it is shameful and just a bad practice to not have the system which allow fast security patch deployed. For a system that holds sensitive personal information for hundreds of millions of people it should a very serious issue not to patch the system for months after vulnerability was known.

Re: Patching is hard; so what?

#10

Post by hyperpape » Sat, Sep 16, 2017, 2:47 AM UTC

guelo wrote: post #7

Patching isn't that hard. Once Equifax finally noticed the intrusion (intrusion detection system?), they took the system down and patched it within a day.

I'm working with a fifteen year old app that's slowly modernizing. There's two types of updates, the normal kind that's slow, and "everything is on fire, get the fix into production ASAP, then pray". The latter kind is super quick, and it's what Equifax did here. But that doesn't mean patching is easy for them. It means patching without doing whatever laborious QA/manual testing they normally do is easy.

(Btw: it might sound like I'm defending Equifax. I'm not. I fully agree with the article. They have a responsibility to figure out how to secure their systems. "We can't patch quickly" may be true for them, but they needed to figure out how to change that).