Summary of June 8 outage
fastly.com
Summary of June 8 outage
1–10 of 110 posts
Re: Summary of June 8 outage
#2So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture,
https://www.cloudflare.com/learning/security/glossary/what-i...
Re: Summary of June 8 outage
#3Wasn't there a failover or some redundency from Fastly in place during this outage?
Re: Summary of June 8 outage
#4So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-i...
Yeah it seems an architecture that even allows a single end-customer issue to take down the entire network may need a bit of rearchitecting.
Re: Summary of June 8 outage
#5This is annoyingly vague. What was the software bug, and what was the valid customer configuration change?
It's perhaps a bit premature to demand it at this point, but I'm hoping a full post-mortem will outline precisely how this change was not picked up in pre-prod. Surely all valid customer configurations must be tested prior to rollout.
Re: Summary of June 8 outage
#6So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-i...
Why do so many big companies use Fastly when Cloudflare (from the outside, as someone who doesn't know much about the space) looks to be so much cleaner and more technically sophisticated? Am I being brainwashed by their blog posts?
Re: Summary of June 8 outage
#7Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration change safety mechanisms/smoke tests/gradual rollout.
Also, what is up with their partitioning? Do they seriously have one customer that gets served from 85% of their servers? Is it a whale?
Good on them for getting a statement out right away (although they basically had to) but seems to be lacking any useful details. Wonder if they were scrubbed by PR/legal in hopes of reducing the number of customers coming to ask for gibs.
Re: Summary of June 8 outage
#8So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-i...
Zero trust seems to be very unrelated to this issue. The issue seems to have been a poison config breaking fastly stack. Zero trust is about verifying authentication of devices/users. Unrelated things really.
Re: Summary of June 8 outage
#9So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-i...
Why do so many big companies use Fastly when Cloudflare (from the outside, as someone who doesn't know much about the space) looks to be so much cleaner and more technically sophisticated? Am I being brainwashed by their blog posts?
Truly different capabilities under the hood. Yes, they are both CDN providers, but fastly offers a remarkable amount of customization that cloudflare does not.
For 99% of customers, one can argue that cloudflare is more than sufficient. For 1% of customers, fastly is arguably the correct choice just based on feature set alone.
So, in summary, you can certainly compare the two, but for certain customers cloudflare lacks the feature set they may choose to use on fastly.
Re: Summary of June 8 outage
#10Let’s just hope that this is just the first statement we see about the outage. This is not even 24 hours after the outage, so I’ll give them the benefit of the doubt that there will be more details forthcoming.
But I can’t help but be bothered that a single customer’s configuration change would have such a wide ranging impact across so many sites. I’m looking forward to finding out how that happens…