Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Patching is hard; so what?

blog.cryptographyengineering.com

Post reply on HN
Print view View original

41–50 of 64 posts

Re: Patching is hard; so what?

#41

Post by Spooky23 »

chronid wrote: post #31

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them. Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy". But if the business does not prioritize (and allocate…

I am responsible for about 200,000 production systems, about 90/10 client to server.

Delivery is not difficult. My teams can deliver anything to 95% of those systems in It's also more effective to understand and have mitigating controls. Many java related exploits cannot be patched with off the shelf software -- you need to wait for the vendor. So you to segment or monitor to live at risk.

Companies who are good at this have good asset management, good process and good vulnerability monitoring.

Re: Patching is hard; so what?

#42

Post by dec0dedab0de »

Spooky23 wrote: post #41
chronid wrote: post #31

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them. Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy". But if the business does not prioritize (and allocate…

I am responsible for about 200,000 production systems, about 90/10 client to server. Delivery is not difficult. My teams can deliver anything to 95% of those systems in It's also more effective to understand and have mitigating controls. Many java related exploits cannot be patched with off the shelf software -- you need to wait for the vendor. So you to segment or monitor to live at risk. Companies who are good at t…

Good fonts?

Re: Patching is hard; so what?

#43

Post by Spooky23 »

dec0dedab0de wrote: post #42
Spooky23 wrote: post #41

Earlier quoted context omitted.

I am responsible for about 200,000 production systems, about 90/10 client to server. Delivery is not difficult. My teams can deliver anything to 95% of those systems in It's also more effective to understand and have mitigating controls. Many java related exploits cannot be patched with off the shelf software -- you need to wait for the vendor. So you to segment or monitor to live at risk. Companies who are good at t…

Good fonts?

Thanks for catching that quickly! I don't remember precisely what word i intended to write... but autocorrect butchered it!

I suppose adding "quality control" to my original post would be appropriate as well! :)

Re: Patching is hard; so what?

#44

Post by ams6110 »

pizza wrote: post #26
Sevii wrote: post #22

Earlier quoted context omitted.

If there is no penalty that hurts shareholders and executives breaches aren't going to stop.

Penalizing shareholders and executives to the extent that reporting breaches becomes enterprise suicide is only going to make breaches unannounced and prevent the collective from learning from individual mistakes, so therefore more common. The way to reduce breaches is to encourage open admission of security failures, so that preventative knowledge spreads quickly. It is worth some positive amount of respect to follo…

That is the air crash investigation model, which has as a primary goal not assigning blame but understanding causes and what can be done differently in the future.

But in many cases it boils down to "pilot error" and in those cases the pilots (if they survived) see their careers severely limited if they even fly at all anymore.

Re: Patching is hard; so what?

#45

Post by watwut »

empath75 wrote: post #2

I know equifax is probably dealing with a hairy legacy system, but really you should design all your systems to be able to push out a fully tested blue/green deploy in hours not months. There's no excuse not to with Jenkins/aws/docker/etc. You should be testing and pushing updates regularly as a matter of course, not as a red alert only when security vulnerabilities are published.

The "change takes time" issue does not happen because of technical problems. It happens because of organizational problems. Technical problems with patching are easiest to solve.

Side note: fully tested within hours assumes small stable software. Large software with frequent changes into it needs to be manually tested. If you have microservices, you still have to test whether something important did not broke.

Re: Patching is hard; so what?

#46

Post by watwut »

ams6110 wrote: post #12
empath75 wrote: post #2

I know equifax is probably dealing with a hairy legacy system, but really you should design all your systems to be able to push out a fully tested blue/green deploy in hours not months. There's no excuse not to with Jenkins/aws/docker/etc. You should be testing and pushing updates regularly as a matter of course, not as a red alert only when security vulnerabilities are published.

That's not the mindset of enterprise IT though. The mindset of enterprise IT is "if it works, don't touch it." This mindset developed because of decades of dealing with arcane, complicated software that was at the same time critical to business operations. I'm aware of a major university that still has a three-week "change freeze" at the start of every semester because of this attiude and I'm sure they are not the on…

Three weeks freeze twice a year during the most critical time does not sound that horrible to me.

Software developers tend to break system with changes. That is not some kind of mythical rare issue - that happens often (with some vendors very often).

Re: Patching is hard; so what?

#47

Post by hyperpape »

sqldba wrote: post #35
hyperpape wrote: post #10

Earlier quoted context omitted.

I'm working with a fifteen year old app that's slowly modernizing. There's two types of updates, the normal kind that's slow, and "everything is on fire, get the fix into production ASAP, then pray". The latter kind is super quick, and it's what Equifax did here. But that doesn't mean patching is easy for them. It means patching without doing whatever laborious QA/manual testing they normally do is easy. (Btw: it mig…

Yes. You've hit the nail on the head. They'll have such a tight bureaucracy on patching that it's impossible (which is an IT problem in their eyes) but the second a senior manager says just do it - its quick and painless. I see this over and over again everywhere.

It's quick, but it's risky, because testing is manual, you don't have blue-green deploys, etc. So it's not that it becomes painless, it's just that their are circumstances where you accept the risk.

Re: Patching is hard; so what?

#48

Post by alkonaut »

sqldba wrote: post #33
chronid wrote: post #31

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them. Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy". But if the business does not prioritize (and allocate…

Those "don't touch" systems are death. It's usually in contracts with the big players like Honeywell and others. You literally can't touch them without breaking the contract. The second you call for support and they find a patch - you have to uninstall before they continue - even if it's unrelated. BTW hands up if that includes WannnaCrypt. Yep.

This is the same situation as those MRI machines with XP on them. The answer is don't buy those machines or don't connect machines you can't patch to the network. If hospitals can't buy them because those are the criteria, then industry will change.

Don't sign contracts that leaves you in the situation that you can't patch machines.

Re: Patching is hard; so what?

#49

Post by alkonaut »

It's pretty simple to patch if you designed the system to be patchable in the first place and assign enormous cost to not patching. If you assume running an outdated version of struts for 3months is a $1B expense then you'll patch even if it means disruption to business operations, or will breach some conract.

And if the system is properly designed then anything can be rolled back if it turns out there were unforeseen consequences.

Re: Patching is hard; so what?

#50

Post by hn_throwaway_99 »

rythie wrote: post #37

Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt it. Systems need to be designed to be patched regularly from the outset. Yes, defense in depth can help, but it's not defense in depth if you one of the layers is pretty much always broken. If they are relying on legacy systems that can't be proactively and regularly be patched, they shouldn't be holding that data. I think part of the probl…

> Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt it.

Companies with monopolistic pricing power like Facebook and Google are a poor comparison for almost any other company. Many problems become very easy when you have a near infinite supply of money. Other businesses need to work with much more difficult constraints.

Post reply on HN