Will managers continue to push engineers even when engineers advise to go slower or no?
Always.
Preliminary Post Incident Review
41–50 of 227 posts
Re: Preliminary Post Incident Review
#42> How Do We Prevent This From Happening Again? > Software Resiliency and Testing > * Improve Rapid Response Content testing by using testing types such as: > * Local developer testing So no one actually tested the changes before deploying?!
And why is it "local developer testing" and not CI/CD. This makes them look like absolute amateurs.
Re: Preliminary Post Incident Review
#43> How Do We Prevent This From Happening Again? > Software Resiliency and Testing > * Improve Rapid Response Content testing by using testing types such as: > * Local developer testing So no one actually tested the changes before deploying?!
And why is it "local developer testing" and not CI/CD. This makes them look like absolute amateurs.
> This makes them look like absolute amateurs.
This applies also to all Architects and CTO's at all these Fortune 500 companies, who allowed these self updating systems into their critical systems.
I would offer a copy of Antifragile to each of these teams: https://en.wikipedia.org/wiki/Antifragile_(book)
"Every captain goes down with every ship"
Re: Preliminary Post Incident Review
#44> How Do We Prevent This From Happening Again? > Software Resiliency and Testing > * Improve Rapid Response Content testing by using testing types such as: > * Local developer testing So no one actually tested the changes before deploying?!
This also becomes a security issue at some point. If these updates can go in untested, what's to stop a rogue employee from deliberately pushing a malicious update?
I know insider threats are very hard to protect against in general but these companies must be the most juicy target for state actors. Imagine what you could do with kernel space code in emergency services, transport infrastructure and banks.
Re: Preliminary Post Incident Review
#45Lots of words about improving testing of the Rapid Response Content, very little about "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes".
> Enhance existing error handling in the Content Interpreter.
That's it.
Also, it sounds like they might have separate "validation" code, based on this; why is "deploy it in a realistic test fleet" not part of validation? I notice they haven't yet explained anything about what the Content Validator does to validate the content.
> Add additional validation checks to the Content Validator for Rapid Response Content. A new check is in process to guard against this type of problematic content from being deployed in the future.
Could it say any less? I hope the new check is a test fleet.
But let's go back to, "the sensor client should not ever count on the Rapid Response Content being well-formed to avoid crashes".
Re: Preliminary Post Incident Review
#46Direct link to the PIR, instead of the list of posts: https://www.crowdstrike.com/blog/falcon-content-update-preli...
Re: Preliminary Post Incident Review
#47There’s only one sentence that matters:
"Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed."
This is where they admit that:
1. They deployed changes to their software directly to customer production machines; 2. They didn’t allow their clients any opportunity to test those changes before they took effect; and 3. This was cosmically stupid and they’re going to stop doing that.
Software that does 1. and 2. has absolutely no place in critical infrastructure like hospitals and emergency services. I predict we’ll see other vendors removing similar bonehead “features” very very quietly over the next few months.
Re: Preliminary Post Incident Review
#48Earlier quoted context omitted.
And why is it "local developer testing" and not CI/CD. This makes them look like absolute amateurs.
> This makes them look like absolute amateurs. This applies also to all Architects and CTO's at all these Fortune 500 companies, who allowed these self updating systems into their critical systems. I would offer a copy of Antifragile to each of these teams: https://en.wikipedia.org/wiki/Antifragile_(book) "Every captain goes down with every ship"
Architects likely do not have a choice. These things are driven by auditors and requirements for things like insurance or PCI and it’s expensive to protest those. I know people who’ve gone full serverless just to lop off the branches of the audit tree about general purpose server operating systems, and now I’m wondering whether anyone is thinking about iOS/ChromeOS for the same reason.
The more successful path here is probably demanding proof of a decent SDLC, use of memory-safe languages, etc. in contract language.
Re: Preliminary Post Incident Review
#49[flagged]
Re: Preliminary Post Incident Review
#50This reads like a bunch of baloney to obscure the real problem. The only relevant part you need to see: >Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data . Problematic content ? Yeah, this is telling exactly nothing. Their mitigation is "ummm we'll test more and maybe not roll the updates to everyone at once", without any direct exp…
Also “using memory safe languages for critical components” and “detecting failures to load and automatically using the last-known-good configuration”