Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Patching is hard; so what?

blog.cryptographyengineering.com

Post reply on HN
Print view View original

11–20 of 64 posts

Re: Patching is hard; so what?

#11

Post by rdtsc »

> Specifically, these folks point out that patching is hard. The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex

Then they should be taken to court and be forced to pay for it. It's like saying about a company which runs trains and didn't do maintenance which caused an accident to happen "Well, you see maintaining the brakes on the trains is too hard so don't blame them too much...".

> The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex as a major framework patch across their production systems.

So those corporations will be hacked, and then they should be dragged to court, made to pay, some might even go bankrupt and disappear altogether. Eventually only the companies that manage to build better infrastructure will survive.

Oh well, one can dream...

Re: Patching is hard; so what?

#12

Post by ams6110 »

empath75 wrote: post #2

I know equifax is probably dealing with a hairy legacy system, but really you should design all your systems to be able to push out a fully tested blue/green deploy in hours not months. There's no excuse not to with Jenkins/aws/docker/etc. You should be testing and pushing updates regularly as a matter of course, not as a red alert only when security vulnerabilities are published.

That's not the mindset of enterprise IT though. The mindset of enterprise IT is "if it works, don't touch it." This mindset developed because of decades of dealing with arcane, complicated software that was at the same time critical to business operations. I'm aware of a major university that still has a three-week "change freeze" at the start of every semester because of this attiude and I'm sure they are not the only one.

Re: Patching is hard; so what?

#13

Post by yeukhon »

chasb wrote: post #3

For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason. If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely. That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.

It isn't necessary DevOps or system admin not capable. You've vendor software not up-to-date that is beyond your control, and often users do not know until vendor sends a notification or a hack headline comes online then "oh fuck" moment.

Just to be fair too: I have seen a number of enterprises run their "legacy" infrastructure (mostly refereing to on-premise servers and old applications) doing a good job keeping up with updates such as Microsoft and whatever Linux flavor they use. But the new system/DevOps havr trouble getting through the gate, mostly IMO because we adding too many layers to the stack.

Re: Patching is hard; so what?

#14

Post by sverhagen »

yeukhon wrote: post #13
chasb wrote: post #3

For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason. If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely. That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.

It isn't necessary DevOps or system admin not capable. You've vendor software not up-to-date that is beyond your control, and often users do not know until vendor sends a notification or a hack headline comes online then "oh fuck" moment. Just to be fair too: I have seen a number of enterprises run their "legacy" infrastructure (mostly refereing to on-premise servers and old applications) doing a good job keeping up…

Your first paragraph, do you find any of that a valid excuse for a company like Equifax, who handle so much sensitive data? _I_ consider them responsible for the entire chain. I understand it's hard, makes no difference with those stakes.

Re: Patching is hard; so what?

#15

Post by topkeker »

ams6110 wrote: post #12
empath75 wrote: post #2

I know equifax is probably dealing with a hairy legacy system, but really you should design all your systems to be able to push out a fully tested blue/green deploy in hours not months. There's no excuse not to with Jenkins/aws/docker/etc. You should be testing and pushing updates regularly as a matter of course, not as a red alert only when security vulnerabilities are published.

That's not the mindset of enterprise IT though. The mindset of enterprise IT is "if it works, don't touch it." This mindset developed because of decades of dealing with arcane, complicated software that was at the same time critical to business operations. I'm aware of a major university that still has a three-week "change freeze" at the start of every semester because of this attiude and I'm sure they are not the on…

I've worked in enterprise security teams for the last decade. This is spot on. It's retarded and will never change because there is no accountability. Corporate IT don't give no fucks.

Re: Patching is hard; so what?

#16

Post by sillysaurus3 »

rdtsc wrote: post #11

> Specifically, these folks point out that patching is hard. The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex Then they should be taken to court and be forced to pay for it. It's like saying about a company which runs trains and didn't do maintenance which caused an accident to happen "Well, you see maintaining the brakes on the trains is too hard so don't b…

The fact that Equifax reported the breach at all represents a good faith understanding: they tell the world, and in return the world does not make them cease to exist.

If you kill companies over security incidents, then magically you will no longer hear about security incidents.

Re: Patching is hard; so what?

#17

Post by zaptheimpaler »

sillysaurus3 wrote: post #16
rdtsc wrote: post #11

> Specifically, these folks point out that patching is hard. The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex Then they should be taken to court and be forced to pay for it. It's like saying about a company which runs trains and didn't do maintenance which caused an accident to happen "Well, you see maintaining the brakes on the trains is too hard so don't b…

The fact that Equifax reported the breach at all represents a good faith understanding: they tell the world, and in return the world does not make them cease to exist. If you kill companies over security incidents, then magically you will no longer hear about security incidents.

They aren't doing us a favor that we have to reciprocate by reporting a breach.. its more of a responsibility. If its not required by law already, it should be.

Re: Patching is hard; so what?

#18

Post by mirashii »

sillysaurus3 wrote: post #16
rdtsc wrote: post #11

> Specifically, these folks point out that patching is hard. The gist of these points is that you can’t expect a major corporation to rapidly deploy something as complex Then they should be taken to court and be forced to pay for it. It's like saying about a company which runs trains and didn't do maintenance which caused an accident to happen "Well, you see maintaining the brakes on the trains is too hard so don't b…

The fact that Equifax reported the breach at all represents a good faith understanding: they tell the world, and in return the world does not make them cease to exist. If you kill companies over security incidents, then magically you will no longer hear about security incidents.

It's not a good faith gesture, it's required by law almost everywhere they operate that they notify of a security breach of PII.

http://www.ncsl.org/research/telecommunications-and-informat...

Re: Patching is hard; so what?

#19

Post by Johnny555 »

rdtsc wrote: post #9
chasb wrote: post #3

For teams that use a DevOps model, fast, predictable deploys that can be safely rolled back are important for security, for this reason. If deploys are like playing Jenga on a sailboat, you're not going to be able to patch fast or safely. That said, even becoming aware a CVE exists in the first place is still a problem for many teams. There are plenty of good options, it's just underinvested in early on.

Exactly. I'd say the check of how fast can a security patch be deployed from the moment upstream OS or library releases is a good benchmark for a team/system and should be evaluated at the same level as log audits, what ports are open and what openssl versions are running. > it's just underinvested in early on. And for cases where there is no personal or financial data involved it is shameful and just a bad practice…

Exactly. I'd say the check of how fast can a security patch be deployed from the moment upstream OS or library releases is a good benchmark for a team/system and should be evaluated at the same level as log audits, what ports are open and what openssl versions are running.

Deploying patches is easy, validating that they aren't going to break anything (sometimes in hard to detect ways) is hard.

Re: Patching is hard; so what?

#20

Post by Johnny555 »

shawkinaw wrote: post #8

Maybe patching is hard. But the real question is why one vuln in their web framework allowed unfettered access to basically all of their data. There are much deeper problems here than slow patching.

the real question is why one vuln in their web framework allowed unfettered access to basically all of their data

The hack involved "primarily names, Social Security numbers, birth dates, addresses and, in some instances, driver's license numbers", that's a very limited subset of the data they hold on people.

I'm not defending them, but this was not "basically all of their data".

Post reply on HN