Earlier quoted context omitted.
> we know how to do it better, but nearly all of even the smartest engineers we can hire are not smart enough. This seems like a contradiction. If the smartest engineers you can hire are not smart enough to work within formal verification constraints then we in fact do not know how to do this. If formal verification hinges on having perfect engineers then it’s useless because perfect engineers wouldn’t need formal ve…
It’s not that we can’t do it, it’s that higher-velocity occasionally buggy code has proven time and time again to be significantly more profitable than formally verified. The juice is rarely worth the squeeze.
Cloudflare outage should not have happened
161–170 of 265 posts
Re: Cloudflare outage should not have happened
#162Re: Cloudflare outage should not have happened
#163> 1. No nullable fiels.
Is that a typo there? fiels should be fields?
Re: Cloudflare outage should not have happened
#164You don't know the context, you don't know _anything_ except for what Cloudflare chooses to share.
There are very few companies who deal with the kind of load that Clouldflare does, I dread to think what weird edges cases they've run into because of their sheer scale.
Re: Cloudflare outage should not have happened
#165Unless you work at Cloudflare or have worked at Cloudflare I'm not sure opinions like this help. You don't know the context, you don't know _anything_ except for what Cloudflare chooses to share. There are very few companies who deal with the kind of load that Clouldflare does, I dread to think what weird edges cases they've run into because of their sheer scale.
Re: Cloudflare outage should not have happened
#166I agree it should not have happened, but I don’t agree that the database schema is the core problem. The “logical single point of failure” here was created by the rapid, global deployment process. If you don’t want to take down all of prod, you can’t update all of prod at the same time. Gradual deployments are a more reliable defense against bugs than careful programming.
>Gradual deployments are a more reliable defense against bugs than careful programming The challenge, as I understand it, is that the feature in question had an explicit requirement of fast, wide deployment because of the need to react in real time to changing external attacker behaviors.
Re: Cloudflare outage should not have happened
#167I was expecting a critique on the centralized nature of the infrastructure and the fragility that comes with it.
I was hoping for a critique of the latter.
Re: Cloudflare outage should not have happened
#168Earlier quoted context omitted.
Are Cloudflare's customers willing to pay avionics software level prices?
Given that Cloudflare's market cap is 1/2 of Boeing's and they are not making a physical product I would say: Clearly, yes.
Re: Cloudflare outage should not have happened
#169Earlier quoted context omitted.
Given that Cloudflare's market cap is 1/2 of Boeing's and they are not making a physical product I would say: Clearly, yes.
Boeing only makes this class of software quality because they are forced to by law. No one does it unless there is a big expensive legal reason to do so.
Re: Cloudflare outage should not have happened
#170Earlier quoted context omitted.
> That's ok, but then you should bow out of the conversation, which is between people that do understand why anyone should want this. I was not making a statement that I am ignorant. I was saying I believe the proposal to model general software engineering after avionics is misguided and inviting you to clarify your position. It is certainly valid to ask what CloudFlare or anyone else for that matter could learn from…
> I was not making a statement that I am ignorant. ok. > I was saying I believe the proposal to model general software engineering after avionics is misguided and inviting you to clarify your position. But we are not talking about 'general software engineering', we are talking about Cloudflare specifically and that makes a massive difference. > It is certainly valid to ask what CloudFlare or anyone else for that matt…
I'm not sure about that. This type of conversation tends toward "shit's easy syndrome" with complexities hand waved away and real trade offs given lip service consideration only. With respect to CloudFlare you specifically said "as soon as they become the cause of an outage they have invalidated their whole reason for existence". I don't know how to square black and white statements like that with an understanding of tradeoffs. A lot of companies would (and do) trade the potential for an outage against the ongoing value of CloudFlare's offerings.
> we are wondering if there would have been better alternatives either by using different constructs or by using different engineering principles.
I think what was actually said was "let's start off with holding them to the same standards as avionics software development". Not so much inquisitive as "shit's easy".
> And there is a pretty good chance that had they done that that they would have come out ahead.
How did you reach that conclusion? CloudFlare has taken a stock hit recently. Even if we attribute that 100% to their outage, they are still up 92% over the last year.
For comparison's sake, CloudFlare was founded after the 737 Max started development. I seriously doubt CloudFlare would have achieved its current success by attempting to ape avionics engineering.
> That is very much not the case. The environment aircraft are subject to are - and increasingly so due to climate change - dynamic to a point that would probably surprise you.
Did you honestly think I was referring to the actual weather? A plane built in 1970 will (assuming it's been maintained) still fly today just fine. The design constraints today are essentially the same and there are no adversaries out there changing the weather in a way that Boeing needs to continuously account for.
This is wholly different from CloudFlare, who is actively fighting botnets and other adversaries who are continuously adapting and changing tactics. The closest analog for avionics would probably be nation states that can scramble GPS.
> In that sense the differences with Cloudflare are in fact not that large.
In the sense that both are important and both happen to involve software, sure. In most other ways the differences are in fact very large.
> I do not believe that you are correct here. They could, they can afford it and they have reached a scale at which the door is firmly closed against competitors, this is not a two bit start-up anymore.
You are ignoring the reality of the situation, and it surfaces in self-contradictory statements like this. They have closed the door firmly on competition so now they need to focus on avionics-like engineering? Why? If their moat is unpassable they should just stop development and keep raking in money. The only reason that they even experienced this outage was because they are in continuous development.
The reality is that their moat is not that wide. If their adversaries or their competition outpace them, they could easily lose their customers to AWS or Azure or someone else.
> This too does not show deep insight into the kind of engineering that goes into any particular bridge. That they look the same to you is just the outside, the interface. But how a particular bridge is anchored and engineered can be a world of a difference from another bridge in a different soil situation, even if they look identical. The big trick is that they all look like simple constructs, but they're not.
Forest for the trees... I did not claim that the bridges are actually the same. But how to build foundations, how to span supports, how thick concrete needs to be and how much rebar, these are well established. Yes, there are calculations and designs but civil engineers have done an excellent job of building a large corpus of practical information that allows them to build bridges with confidence. (And this is definitely something we could learn from them.) Rarely are bridges built mostly with custom components that have never been used before.
> Boeing fucked up, not some software engineer taking a short-cut. This was a top down managed disaster with multiple attempts to cover up the root cause and a complete failure of regulatory oversight.
You're trying to hand wave this away as if I am blaming some individual Boeing engineer, but I'm not.
Engineering isn't just coding. Engineering is the planning and the designing and the building and the testing and everything else that makes the product what it is. Boeing created a system to mask the flight characteristics of their new plane, except it didn't actually work. (And also yes they lied to regulators about it.) If it actually worked it those two planes wouldn't have crashed. A product intended to make planes easier to fly is poorly engineered if it actually crashes planes.