Live data from Hacker News

Patching is hard; so what?

blog.cryptographyengineering.com

51–60 of 64 posts

Re: Patching is hard; so what?

#51

Earlier quoted context omitted.

It's not a good faith gesture, it's required by law almost everywhere they operate that they notify of a security breach of PII. http://www.ncsl.org/research/telecommunications-and-informat...

And yet, as if by magic, you will no longer hear about security incidents if you kill companies.

How about this: You have a vulnerability and you disclose, your company ceases to exist.

You have a vulnerability and you fail to disclose, your company ceases to exist and all your C-level execs go to jail.

Re: Patching is hard; so what?

#52
post #31

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them. Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy". But if the business does not prioritize (and allocate…

> with B/G deployments

Why do we call it that now, instead of A/B deployment like we used to?

Re: Patching is hard; so what?

#53
post #52
post #31

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them. Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy". But if the business does not prioritize (and allocate…

> with B/G deployments Why do we call it that now, instead of A/B deployment like we used to?

To me, blue/green is for infrastructure, and A/B is for the user side of it. That is, you should be able to do blue/green with the content not changing, but when you're A/B testing, you're collecting metrics and other usage data about the users to make future product decisions. You can also do A/B testing without having a B/G setup, using things like user gates. This distinction may be incorrect, but that's how I've used them.

Re: Patching is hard; so what?

#54
post #52
post #31

> I don’t operate production systems, but I have helped to design a couple of them. So I understand something about the assumptions you make when building them. Start by operating production systems. You will rapidly discover that patching is not a technical issue (as I already said last time there was a patching discussion on HN). Patching is technically "easy". But if the business does not prioritize (and allocate…

> with B/G deployments Why do we call it that now, instead of A/B deployment like we used to?

A/B implies certain symmetry, while B/G is highly asymmetrical: G is known good, battle-tested, and B is new and suspicious.

Re: Patching is hard; so what?

#55
post #7

Patching isn't that hard. Once Equifax finally noticed the intrusion (intrusion detection system?), they took the system down and patched it within a day.

I'm working with a fifteen year old app that's slowly modernizing. There's two types of updates, the normal kind that's slow, and "everything is on fire, get the fix into production ASAP, then pray". The latter kind is super quick, and it's what Equifax did here. But that doesn't mean patching is easy for them. It means patching without doing whatever laborious QA/manual testing they normally do is easy. (Btw: it mig…

> It means patching without doing whatever laborious QA/manual testing they normally do is easy.

Everyone learns how to do Continuous Deployment when prod is on fire.

Re: Patching is hard; so what?

#56
"I don’t dispute this point. It’s absolutely valid."

I do dispute it. Patching is not hard. It's incredibly easy to the point that some places and projects have the process automated from notification to application to testing for breakage to deployment. For reducing downtime, one can use clusters with rolling deployments of patches. Patching is only hard for things such as web applications when the company's application or processes are done in a way that makes patching hard. As in, they have to be incompetent or just not care about IT. Far as competence, your claim about fragile systems is a good example that might have happened.

" then you’ve implicitly made the decision that you’re never ever going to allow those vulnerabilities to fester."

You're thinking like an engineer that cares about quality. You instead should think like an Equifax CIO or something. To start with, this is a company that collects PII against people's will to sell to third parties for their main goal of huge profits. Politics plays more a role than engineering talent in people getting the senior, management positions in such companies. They also tend to chase whatever is popular among Fortune 500, esp with cheap labor or ecosystems available. Java was one of the fads that financial sector was all over. Combine all this to have a company whose fad-chasing CIO keeps costs down and profits up applying the thing he or she read in a computer magazine with the cheapest talent available on a tight budget. The result of their work is a pile of garbage they have trouble patching. If you doubt this, just look at the security of the web site they deployed for credit monitoring and apparently to help hackers get at people interested in credit monitoring. Or they just made mistakes so easily avoided that they're either inexperienced beginners or don't care at all.

"So what would those systems look like?"

Well, they would have built it some time ago. So, let's work our way from old, high-assurance security toward something commercial and affordable from at least 2000-2005 era. The original work in securing data involved security kernels:

https://www.acsac.org/secshelf/book001/19.pdf

Several of those are still available but expensive. Both security kernels and databases such as Trusted Rubix built for them. Today, those look more like the next link with companies such as Sirrix and Green Hills selling them commercially:

https://os.inf.tu-dresden.de/papers_ps/nizza.pdf

In any case, we'd need a robust combination of OS, database, and application code. Nothing hits the database without going through the app server first. So, we embed our security policy into app server. How to implement it? Ever since Dijkstra's THE OS (1960's), we knew to specify correct behavior with preconditions, invariants, and post conditions. Then use anything from formal analysis to testing to runtime checks to ensure that behavior is enforced. The security kernels did the former where Design-by-Contract in Eiffel used tests and runtime checks. Got to pick method with best bang for buck. Two links to illustrate safer languages of past with Ada 2005 since we're looking at earlier stuff.

http://www.adacore.com/knowledge/technical-papers/safe-secur...

https://www.eiffel.com/values/design-by-contract/introductio...

Given labor situation, we use DbC in a safe language. All prior work showed simplicity was necessary for security. So, it would be an app server done in Eiffel, Ada, or (if absolutely necessary) Java/C++ using Design-by-Contract methodology plus lots of testing. The web component would be a middleware done similarly that basically translates web actions to simple protocol the server uses for requests/responses. Any new types of problems with web apps are mitigated in the simple framework a la Airship CMS, caught with some kind of monitoring a la Spectre, or both.

This would have to run on something robust. You'd have the database/storage, app server, web component, management, and monitoring which is behind one-way link (data diode) w/ logs copied to write-only media. A company from likely mainframe background now run by wise technologists might first make the software target IBM but portable. Balancing reliability, security, and cost, the best targets would be AS/400 or AIX. The former is a capability-oriented architecture with few 0-days whereas other is a rock-solid UNIX. Portability means they'd migrate part or all of this to new OS's as they showed up. Eventually they'd notice OpenBSD's security advantages or buy one of those certified-secure kernels w/ POSIX layer. Depends on their viewpoint and budget. Probably throw some clustering software with rolling releases in there since OpenVMS and NonStop had high-availability in the 1980's. It was well-known strategy.

So, there was a straight-forward way to make a robust stack with a web front end which gets more robust over time. This could be done in high availability or just easy patches. They instead end up with fad-driven crud possibly running on other crud that's easy to hack but hard to patch. Just bad engineering that's typical in the market versus methods such as I described. Also note that there were (are) large institutions using some of what I recommended for those benefits. The tools and people just cost a bit more which doesn't align with incentives of companies as greedy and IT hating as Equifax. Sure turned out extra profitable, didn't it? ;)

Re: Patching is hard; so what?

#57
post #13

Earlier quoted context omitted.

It isn't necessary DevOps or system admin not capable. You've vendor software not up-to-date that is beyond your control, and often users do not know until vendor sends a notification or a hack headline comes online then "oh fuck" moment. Just to be fair too: I have seen a number of enterprises run their "legacy" infrastructure (mostly refereing to on-premise servers and old applications) doing a good job keeping up…

Your first paragraph, do you find any of that a valid excuse for a company like Equifax, who handle so much sensitive data? _I_ consider them responsible for the entire chain. I understand it's hard, makes no difference with those stakes.

No, not at all. My point was DevOps is not a magical group of wizards, and often DevOps team can be unproductive compared to whatever enterprise IT operation team/client service team. If DevOps is unable to carry out its core values, yes, entite chain is to be critized.

Re: Patching is hard; so what?

#58
post #36
post #23

Earlier quoted context omitted.

There exists some space between no penalty and a penalty that effectively kills the company.

Besides, there exists the level of mismanagement which deserves to tank the company. Investment in security costs money. Ignoring it saves money and gives company advantage against competitors who invested into it - untill breach happens. If we then effectively ok breach, we practically ensured that only bad security companies will remain living in the long term.

Taleb called this "picking up pennies in front of a steamroller". You make a little money every day, until you get killed. Lots of management consultants specialize in recommending this stupid practice, because corporate executives can't get enough of it. It doesn't seem likely we'll see a law that effectively punishes firms for this behavior in the PII security context, because the same behavior in other contexts led not to failing firms but rather to "too big to fail" and bailouts.

Re: Patching is hard; so what?

#59

Earlier quoted context omitted.

I'm working with a fifteen year old app that's slowly modernizing. There's two types of updates, the normal kind that's slow, and "everything is on fire, get the fix into production ASAP, then pray". The latter kind is super quick, and it's what Equifax did here. But that doesn't mean patching is easy for them. It means patching without doing whatever laborious QA/manual testing they normally do is easy. (Btw: it mig…

> It means patching without doing whatever laborious QA/manual testing they normally do is easy. Everyone learns how to do Continuous Deployment when prod is on fire.

If you're not managing the risk of upgrades, you're arguably not really doing Continuous Deployment.

Re: Patching is hard; so what?

#60
post #33

Earlier quoted context omitted.

Those "don't touch" systems are death. It's usually in contracts with the big players like Honeywell and others. You literally can't touch them without breaking the contract. The second you call for support and they find a patch - you have to uninstall before they continue - even if it's unrelated. BTW hands up if that includes WannnaCrypt. Yep.

This is the same situation as those MRI machines with XP on them. The answer is don't buy those machines or don't connect machines you can't patch to the network. If hospitals can't buy them because those are the criteria, then industry will change. Don't sign contracts that leaves you in the situation that you can't patch machines.

If a hospital doesn't have an MRI machine, it won't be able to compete. It's really not an option to say "I won't buy it unless X." You buy the one that meets most of your needs for the best price. Not buying is not really a choice.
Post reply on HN