541–550 of 560 posts
I don't want to criticize cloud flare, I love what they do and understand the scale of the challenge, but most people don't and 2 in a month or so like this is going to hit their reputation.
Earlier quoted context omitted.
I’m really curious what their rollout procedure is, because it seems like many of their past outages should have been uncovered if they released these configuration changes to 1% of global traffic first.
They don't appear to have a rollout procedure for some of their globally replicated application state. They had a number of major outages over the past years which all had the same root cause of "a global config change exposed a bug in our code and everything blew up". I guess it's an organizational consequence of mitigating attacks in real time, where rollout delays can be risky as well. But if you're going to do th…
Earlier quoted context omitted.
For hypothetical conflicting changes (read worst case: unupgraded nodes/services can't interop with upgraded nodes/services), what's best practice for a partial rollout? Blue/green and temporarily ossify capacity? Regional?
- Push a version with the new logic but not yet enabled, still using legacy logic, able to implement both - Push a version that enables new logic for 1% of traffic - Continue rollout until 100%
This is not good. One major outage? Something exceptional. Several outages in a short time? As someone thats worked in operations, I have empathy; there are so many “temp havks” that are put in place for incidents. but the rest of the world won’t… they’re gonna suffer a massive reputation loss if this goes on as long as the last one.
This will be another post-mortem of...config file messed...did not catch...promise to be doing better next....We are sorry. They problem is architectural.
it will randomly fail. there is no way it cannot.
there is a point where the cost to not fail simply becomes too high.
Earlier quoted context omitted.
Here is the Java equivalent of what happened in that Cloudflare Rust code: try { data = loadDataFile(); } catch (Exception e) { LOG.error("Failed to load new data file", e); System.exit(1); } So the "bad data load" was trapped, but the programmer decided that either it would never actually occur, or that it is unrecoverable, so it is fine to .unwrap(). It would not be any less idiomatic if, instead of crashing, the p…
Yes, I know. But nobody writes code like that in Java. I don't think I've ever seen it outside of top level code in CLI tools. Never in servers. > It is that programmer's fault, and has nothing to do with Rust. It's Rust's fault. It provides a function in its standard library that's widely used and which aborts the process. There's nothing like that in the stdlibs of Java or .NET > Also, if you use general try-catch…
It is the same as runtime exceptions in Java. In Rust, if you want to have a top-level "exception handler" that catches everything, you can do
::std::panic::catch_unwind(|| {
// ...
})
Earlier quoted context omitted.
- Push a version with the new logic but not yet enabled, still using legacy logic, able to implement both - Push a version that enables new logic for 1% of traffic - Continue rollout until 100%
Can also do canary rollout before that. Canary means rollout to endpoints only used by CF to test. Monitor metrics and automated test results.
Earlier quoted context omitted.
Can also do canary rollout before that. Canary means rollout to endpoints only used by CF to test. Monitor metrics and automated test results.
That's ok but doesn't solve issues you notice only on actual prod traffic. While it can be a nice addition to catch issues earlier with minimal user impact, best practice on large scale systems still requires a staged/progressive prod rollout.
Unit test, Integration Test, Staging Test, Staging Rollout, Production Test, Canary, Progressive Rollout
Can all be automated can smash through all that quickly with no human intervention.
Earlier quoted context omitted.
As bad as (3) sounds, I'll strongman the argument: it's important to keep the economic cost of any regulation in mind.* On the one hand, you'd like to prevent the thing the regulation is seeking to prevent. On the other hand, you'd have costs for the regulation to be implemented (one-time and/or ongoing). "Is the good worth the costs?" is a question worth asking every time. (Not least because sometimes it lets you do…
I think conventionally the verb is "to steelman" with the intended contrast being to a strawman, an intentionally weak argument by analogy to how straw isn't strong but steel is. I understood what you meant by "strongman" but I think that "steelman" is better here. There is indeed a good reason regulators aren't just obliged to institute all recommendations - that would be a lot of new rules. The only accident report…
Difficult, but not impossible.
What are calculable and do NOT scale down is cost for compliance documentation and processes. Changing from 1 form of documentation to 4 forms of documentation has measurable cost, that will be imposed forever.
> It's also difficult to anticipate all benefits from a good change without trying it.
That's not a great argument, because it can be counterbalanced by the equally true opposite: it's difficult to anticipate all downsides to a change without trying it.
> Remember when you were allowed to smoke cigarettes on aeroplanes?
Remember when you could walk up to a gate 5 minutes before a flight, buy a ticket, and fly?
The current TSA security theater has had some benefits, but it's also made using airports far worse as a traveler.
Earlier quoted context omitted.
This is a good reminder for everyone to reconsider making all of their websites depend on a single centralized point of failure. There are many alternatives to the different services which Cloudflare offers.
But the nature of a CDN and most other products CF offers, is central by nature. If you switch from CF to the next CF competitor, you've not improved this dependency. The alternative here, is complex or even non-existing. Complex would be some system that allows you to hotswap a CDN, or to have fallback DDOS protection services, or to build you own in-house. Which, IMO, is the worst to do if your business is elsewher…