Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Patching is hard; so what?

blog.cryptographyengineering.com

Post reply on HN
Print view View original

31–40 of 64 posts

Re: Patching is hard; so what?

#31

Post by chronid »

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them.

Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy".

But if the business does not prioritize (and allocate time towards) patching (and testing that systems work with after) patching won't get done. There are features to develop and deadlines to meet and new systems to turn on and old systems to retire and oh-my-god don't touch those servers otherwise the vendor will not support the prod environment anymore (no longer certified, yay!) and it costs us 30000$ to get certified again.

Too many times I see HN assuming that everyone is running on a public cloud with B/G deployments and immutable infrastructure you can rebuild and redeploy easily. Unfortunately that's not the case.

Re: Patching is hard; so what?

#32

Post by sqldba »

Patching at an OS level (not what the author is talking about, probably) is extremely extremely hard - from an enterprise Microsoft platform production point of view

1) Because there's no budget for tools. It's mostly WSUS and SCCM if you're lucky.

2) Because staff don't even know how to use those tools properly. Can't blame them because there's very poor documentation from MS and very little accessible training on design patterns that integrate them into large corps.

3) It's outsourced. So it's even worse than above.

4) Because management fights it every step of the way. They hate patching because they're stuck in the 80s where they think every patch is deeply documented and you should only apply what's necessary. And you should only apply security - ignoring updates which give critical stability fixes.

5) Because the enterprise as a whole is so poorly documented it's unclear who owns what or if a rollback is possible and what it would affect. So everything is in CYA overdrive.

6) And then nobody working there wants to own it because it's not sexy to patch. (As a geek I think patching is sexy and very fulfilling work, but I'm in the extreme minority).

None of this excuses anything. But most largish companies are still in the 80s, with IT managers in the 80s. I can't wait for them all to die.

Re: Patching is hard; so what?

#33

Post by sqldba »

chronid wrote: post #31

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them. Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy". But if the business does not prioritize (and allocate…

Those "don't touch" systems are death. It's usually in contracts with the big players like Honeywell and others. You literally can't touch them without breaking the contract. The second you call for support and they find a patch - you have to uninstall before they continue - even if it's unrelated.

BTW hands up if that includes WannnaCrypt. Yep.

Re: Patching is hard; so what?

#34

Post by sqldba »

rdtsc wrote: post #11

> Specifically, these folks point out that patching is hard. The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex Then they should be taken to court and be forced to pay for it. It's like saying about a company which runs trains and didn't do maintenance which caused an accident to happen "Well, you see maintaining the brakes on the trains is too hard so don't b…

I know people who work with trains. They have truly awesome mechanical systems but cobbled together to IT systems they weren't qualified to program and as awful as you'd imagine and more - undocumented and unpatched.

On the other hand management takes a complete 180 view to their stuff because a train derailment is extremely serious from a profit point of view, costing into the millions, and not so rare that they've never seen one.

It ends up in a real weird jumble of still not acknowledging or patching anything but also being super serious about everything else like the hardware and train maintenance.

I guess it's easy to point at a failed break and say that's the cause but WannaCrypt didn't cause any trains to derail.

Re: Patching is hard; so what?

#35

Post by sqldba »

hyperpape wrote: post #10
guelo wrote: post #7

Patching isn't that hard. Once Equifax finally noticed the intrusion (intrusion detection system?), they took the system down and patched it within a day.

I'm working with a fifteen year old app that's slowly modernizing. There's two types of updates, the normal kind that's slow, and "everything is on fire, get the fix into production ASAP, then pray". The latter kind is super quick, and it's what Equifax did here. But that doesn't mean patching is easy for them. It means patching without doing whatever laborious QA/manual testing they normally do is easy. (Btw: it mig…

Yes. You've hit the nail on the head. They'll have such a tight bureaucracy on patching that it's impossible (which is an IT problem in their eyes) but the second a senior manager says just do it - its quick and painless.

I see this over and over again everywhere.

Re: Patching is hard; so what?

#36

Post by watwut »

KGIII wrote: post #23
Sevii wrote: post #22

Earlier quoted context omitted.

If there is no penalty that hurts shareholders and executives breaches aren't going to stop.

There exists some space between no penalty and a penalty that effectively kills the company.

Besides, there exists the level of mismanagement which deserves to tank the company.

Investment in security costs money. Ignoring it saves money and gives company advantage against competitors who invested into it - untill breach happens. If we then effectively ok breach, we practically ensured that only bad security companies will remain living in the long term.

Re: Patching is hard; so what?

#37

Post by rythie »

Is it so hard though? Do Facebook, Google etc. have this problem too? I doubt it.

Systems need to be designed to be patched regularly from the outset. Yes, defense in depth can help, but it's not defense in depth if you one of the layers is pretty much always broken.

If they are relying on legacy systems that can't be proactively and regularly be patched, they shouldn't be holding that data.

I think part of the problem here, is that we can't see the risks companies are running with our data and goverment doesn't regulate it.

P.S. I do manage systems, though not on this scale.

Re: Patching is hard; so what?

#38

Post by toast0 »

topkeker wrote: post #15
ams6110 wrote: post #12

Earlier quoted context omitted.

That's not the mindset of enterprise IT though. The mindset of enterprise IT is "if it works, don't touch it." This mindset developed because of decades of dealing with arcane, complicated software that was at the same time critical to business operations. I'm aware of a major university that still has a three-week "change freeze" at the start of every semester because of this attiude and I'm sure they are not the on…

I've worked in enterprise security teams for the last decade. This is spot on. It's retarded and will never change because there is no accountability. Corporate IT don't give no fucks.

Corporate IT is doing what software developers have taught them through experience and pavlovian condition:

Don't touch anything, because it will break if you touch it.

IFF people who release software learn how to release security patches and bugfixes that don't include new features or break existing parts (except when breaking changes are actually needed), then, after several years of good results from pushing changes, Corporate IT will be conditioned to just push security fixes, it's not a big deal, plus it's important.

Re: Patching is hard; so what?

#39

Post by empath75 »

ams6110 wrote: post #12
empath75 wrote: post #2

I know equifax is probably dealing with a hairy legacy system, but really you should design all your systems to be able to push out a fully tested blue/green deploy in hours not months. There's no excuse not to with Jenkins/aws/docker/etc. You should be testing and pushing updates regularly as a matter of course, not as a red alert only when security vulnerabilities are published.

That's not the mindset of enterprise IT though. The mindset of enterprise IT is "if it works, don't touch it." This mindset developed because of decades of dealing with arcane, complicated software that was at the same time critical to business operations. I'm aware of a major university that still has a three-week "change freeze" at the start of every semester because of this attiude and I'm sure they are not the on…

Right. Which is why ci/cd and devops, etc were invented. If you're not patching and updating frequently, you're committing business malpractice.
Post reply on HN