Adding distinct or group by to a query is not some advanced technic comments are suggesting. It does not slow down development one bit, if you expect distinct result you put explicit distinct in the query, it's not a "safety measure for insulin pumps". Scratching my head what I've missed here, please enlighten me.
Cloudflare outage should not have happened
101–110 of 265 posts
Re: Cloudflare outage should not have happened
#102Earlier quoted context omitted.
Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…
I’ve been seeing you blazing this trail since the incident and it feels short sighted and reductive. Rust is built on forcing the developer to acknowledge the complexity of reality. Unwrap acknowledges said complexity with a perfectly valid decision. There are a few warts from early days like indexing and the ”as” operator where the easy path is doing the wrong thing. But unwraps or expects are where Rust shines. Thr…
> I’ve been seeing you blazing this trail since the incident and it feels a short sighted and reductive.
Why is it inappropriate to be able to statically label the behavior?
Maybe I don't want my failure behavior dictated by a downstream dependency or distracted engineer.
The subject of how to fail is a big topic and is completely orthogonal to the topic of how can we know about this and shape our outcomes.
I would rather the policy be encoded with first class tools rather than engineering guidelines and runbooks. Let me have some additional control at what looks like to me not a great expense.
It doesn't feel "safe" to me to assume the engineer meant to do exactly this and all of the upstream systems accounted for it. I would rather the code explicitly declare this in a policy we can enforce, in an AST we can shallowly reason about.
Re: Cloudflare outage should not have happened
#103"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…
Software people, especially coming through Rust, are falling into the old trap of believing if code is bug free it is reliable: it isn’t because there is a world of faults outside, including but not limited to the developer intentions. This inverts everything because structuring to be fault tolerant, of the right things, changes what is a good idea almost entirely.
Rust generally forces you to acknowledge these faults. The problem is managing them in a sane way, which for Rust in many cases simply is failing loudly.
Compared to than many other languages which preferring chugging along and hoping that no downstream corruption happens.
Re: Cloudflare outage should not have happened
#104> I base my paragraph on their choice of abandoning PostgreSQL and adopting ClickHouse(Bocharov 2018). The whole post is a great overview on trying to process data fast, without a single line on how to garantee its logical correctness/consistency in the face of changes.
I'm completely mystified how the author concludes that the switch from PostgreSQL to ClickHouse shows the root of this problem.
1. If the point is that PostgreSQL is somehow more less prone to error, it's not in this case. You can make the same mistake if you leave off the table_schema in information_schema.columns queries.
2. If the point is that Cloudflare should have somehow discovered this error through normalization and/or formal methods, perhaps he could demonstrate exactly how this would have (a) worked, (b) been less costly than finding and fixing the query through a better review process or testing, and (c) avoided generating other errors as a side effect.
I'm particularly mystified how lack of normalization is at fault. ClickHouse system.columns is normalized. And if you normalized the query result to remove duplicates that would just result in other kinds of bugs as in 2c above.
Edit: fix typo
Re: Cloudflare outage should not have happened
#105I'd be wanting to have some sort of a "dry run" on the produced artifact by the rust code consuming it, or a deploy to some sort of a test environment before letting it roll out to production. I've been surprised that no mention of that sort of thing in the Cloudflare after-action or here.
Re: Cloudflare outage should not have happened
#106Earlier quoted context omitted.
Agreed. I left out any commentary on `.unwrap()` from my original comment, but it’s an obvious example of something that should never have appeared in critical code.
Rust needs to get rid of .unwrap() and its kin. They're from pre-1.0 Rust, before many of the type system features and error handling syntax sugar were added. There's no reason to use them as the language provides lots of safer alternatives. If you do want to trigger a panic, you can, but I'd also ask - why? Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other…
> There's no reason to use [panics] as the language provides lots of safer alternatives.
Dunno ... I think runtime assertions and the ability to crash a misbehaving program are a pretty important part of the toolset. If rust required `Result`s to be wired up up and down the entire call tree for the privilege of using a runtime assertion, I think it would be a lot less popular, and probably less safe in practice.
> Alternatively, and perhaps even better, Rust needs a way to mark functions that can panic for any reason other than malloc failures.
I 100% agree that a mechanism to prove that code can or cannot panic would be great, but why would malloc be special here? Folks who are serious about preventing panics will generally use `no-std` in order to prevent malloc in the first place.
Re: Cloudflare outage should not have happened
#107"If they had a perfectly normalized database, no NULLing and formally verified code, this bug would not have happened." That may be. What's not specified there is the immense, immense cost of driving a dev org on those terms. It limits, radically, the percent of engineers you can hire (to those who understand this and are willing to work this way), and it slows deployment radically. Cloudflare may well need to transi…
What I have seen work in the past is testing using a production backup as a final step prior to releasing, including applying database scripts. In this case, the permissions change would have been executed, the query would have run, and the failure would have been observed.
Re: Cloudflare outage should not have happened
#108Earlier quoted context omitted.
I’ve been seeing you blazing this trail since the incident and it feels short sighted and reductive. Rust is built on forcing the developer to acknowledge the complexity of reality. Unwrap acknowledges said complexity with a perfectly valid decision. There are a few warts from early days like indexing and the ”as” operator where the easy path is doing the wrong thing. But unwraps or expects are where Rust shines. Thr…
> I’ve been seeing you blazing this trail since the incident and it feels a short sighted and reductive. Why is it inappropriate to be able to statically label the behavior? Maybe I don't want my failure behavior dictated by a downstream dependency or distracted engineer. The subject of how to fail is a big topic and is completely orthogonal to the topic of how can we know about this and shape our outcomes. I would r…
How deep do you go? Being forced to label any function that allocates memory with ”panic”?
Right now you all the instances where the code can panic are labeled. Grep for unwrap, panic, expect etc.
In all my years of professional Rust development I’ve never seen a potential panic pass code review without a discussion. Unless it was trivial like trying to build an invalid Regex from a static string.
Re: Cloudflare outage should not have happened
#109Earlier quoted context omitted.
The vast majority of Cloudflare's "customers" are paying 0 to 20 dollars a month, for virtually the same protection coverage and features as most of their 200 dollars/mo customers. That's not remotely in the realm of avionics price structure, be it software or hardware.
It is the aggregate they pay that counts here, not the individual payments. A better comparison would be to compare this to airline passengers paying for their tickets, they pay a few hundred bucks in the expectation that they will arrive at their destination. Besides, it is not the customers that determine Cloudflare's business model, Cloudflare does. Note that their whole business is to prevent outages and that as…
> as soon as they become the cause of an outage they have invalidated their whole reason for existence
This is a bar no engineering effort has ever met. “If you ever fail, even for a moment, there’s no reason for you to even exist.”
There have been 6 fatal passenger airplane crashes in the US this year alone. NASA only built 6 shuttles and 2 of those exploded, killing their crews. And these were life-preserving systems that failed.
Discussions around software engineering quality always seem to veer into spaces where we assign almost mythic properties to other engineering efforts in an attempt to paint software engineering as lazy or careless.
Re: Cloudflare outage should not have happened
#110While this blog post is pretty useless, it's a hell of a lot better than the LinkedIn posts about the outage... my god, I wish the "Not interested" button worked.