Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of June 8 outage

fastly.com

Post reply on HN
Print view View original

21–30 of 110 posts

Re: Summary of June 8 outage

#21

Post by EE84M3i »

plasma wrote: post #20
EE84M3i wrote: post #7

Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration change safety mechanisms/smoke tests/gradual rollout. Also, what is up with their partitioning? Do they seriously have one customer that gets served from 85% of their servers? Is it a whale? Good on them for getting a statement out right away (although they basical…

At a guess, perhaps each server can serve everyone, and the system-wide config file that is shipped to each server (to handle each customer) perhaps became corrupted as part of a customer update, and it was rolled out to every node assumed to be correct. A defense against this could be to ensure the system that applies the change validates some health-checks continue to work after the new file is made (or automatical…

Yes, it sounds sort of like that. But this can be remediated by two things I was asking about: smoke tests and partitioning.

When making a config change I'd assume they don't make it to all servers at once and instead roll it out gradually. If this caused the server to instantly start 503'ing all customers, presumably this would have been caught - perhaps it was more delayed though (resource leak, etc) and obviously that is somewhat more difficult to catch.

If they're properly partitioning customers, ideally they wouldn't even ship the configs to all servers (slightly less good, but still pretty good they could ship them there but not parse/load them). It sounds like at the least this customer's config change effected 85% of servers, which seems absurd to me.

So yes, I can see how it happened, but for Fastly, which runs one of the biggest CDNs, these don't seem like very reasonable mistakes.

Re: Summary of June 8 outage

#22

Post by thethethethe »

fafner wrote: post #19
thethethethe wrote: post #15

Earlier quoted context omitted.

> commitment to ensure that customer configurations cannot break the whole system You can't just ensure a config change won't break things in large distributed systems, it's too complex with too many factors, there will always be risk. To mitigate your risk, youd want to design your system to do progressive, regional rollouts, with canaries to attempt to detect and isolate before a wide spread outage occurs. Even if…

Unfortunately some customer changes will need to go out quickly and globally for a CDN. Going much slower might not be a good option. Canarying should detect this. Not clear if they do this or the canary failed to report this. Sharding by customers could help reduce blast radius. But maybe not by much of this was a very big customer.

> Unfortunately some customer changes will need to go out quickly and globally for a CDN

Why is this the case? I don't have too much knowledge of CDN architecture so I am curious

Re: Summary of June 8 outage

#23

Post by unityByFreedom »

fafner wrote: post #14
unityByFreedom wrote: post #11

Earlier quoted context omitted.

It's much more than that. It's a whole approach to architecture. The post I linked says it best.

Authentication/trust is simply unrelated to this outage. The post you linked seems to have no relevance.

One of Cloudflare's top engineers previously wrote in this forum,

> This incident emphasizes the importance of the Zero Trust model that Cloudflare follows and provides to customers, which ensures that if any one system or vendor is compromised, it does not compromise the entire organization. [1]

Authentication is a part of a zero-trust model, not the whole thing.

> No single specific technology is associated with zero trust architecture; it is a holistic approach to network security that incorporates several different principles and technologies. [2]

[1] https://news.ycombinator.com/item?id=26407364

[2] https://www.cloudflare.com/learning/security/glossary/what-i...

Re: Summary of June 8 outage

#24

Post by ascorbic »

wjossey wrote: post #9
darwinwhy wrote: post #6

Earlier quoted context omitted.

Why do so many big companies use Fastly when Cloudflare (from the outside, as someone who doesn't know much about the space) looks to be so much cleaner and more technically sophisticated? Am I being brainwashed by their blog posts?

Truly different capabilities under the hood. Yes, they are both CDN providers, but fastly offers a remarkable amount of customization that cloudflare does not. For 99% of customers, one can argue that cloudflare is more than sufficient. For 1% of customers, fastly is arguably the correct choice just based on feature set alone. So, in summary, you can certainly compare the two, but for certain customers cloudflare lac…

Yes, to the extent that Fastly lets you upload your own VCL configuration files. This was the source of the problem here, but is incredibly powerful for complex use cases.

Re: Summary of June 8 outage

#25

Post by rblatz »

thethethethe wrote: post #22
fafner wrote: post #19

Earlier quoted context omitted.

Unfortunately some customer changes will need to go out quickly and globally for a CDN. Going much slower might not be a good option. Canarying should detect this. Not clear if they do this or the canary failed to report this. Sharding by customers could help reduce blast radius. But maybe not by much of this was a very big customer.

> Unfortunately some customer changes will need to go out quickly and globally for a CDN Why is this the case? I don't have too much knowledge of CDN architecture so I am curious

Pushing out new versions of your site. You can’t have the new assets on half the nodes that are serving your site otherwise your site goes down while things slowly propagate.

Re: Summary of June 8 outage

#26

Post by aliasEli »

fafner wrote: post #19
thethethethe wrote: post #15

Earlier quoted context omitted.

> commitment to ensure that customer configurations cannot break the whole system You can't just ensure a config change won't break things in large distributed systems, it's too complex with too many factors, there will always be risk. To mitigate your risk, youd want to design your system to do progressive, regional rollouts, with canaries to attempt to detect and isolate before a wide spread outage occurs. Even if…

Unfortunately some customer changes will need to go out quickly and globally for a CDN. Going much slower might not be a good option. Canarying should detect this. Not clear if they do this or the canary failed to report this. Sharding by customers could help reduce blast radius. But maybe not by much of this was a very big customer.

Sharding would also introduce additional complexity into the system, which is also risky.

Re: Summary of June 8 outage

#27

Post by johncolanduoni »

unityByFreedom wrote: post #23
fafner wrote: post #14

Earlier quoted context omitted.

Authentication/trust is simply unrelated to this outage. The post you linked seems to have no relevance.

One of Cloudflare's top engineers previously wrote in this forum, > This incident emphasizes the importance of the Zero Trust model that Cloudflare follows and provides to customers, which ensures that if any one system or vendor is compromised, it does not compromise the entire organization. [1] Authentication is a part of a zero-trust model, not the whole thing. > No single specific technology is associated with ze…

They were referring to a completely different incident, involving compromised authentication to a camera system. I’d love to hear an explanation of how a zero-trust model would apply to this situation with Fastly. Seems like it would have to apply to a lot of multi-tenant resource exhaustion issues since we know so little about the specifics on the Fastly incident.

Re: Summary of June 8 outage

#28

Post by unityByFreedom »

EE84M3i wrote: post #7

Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration change safety mechanisms/smoke tests/gradual rollout. Also, what is up with their partitioning? Do they seriously have one customer that gets served from 85% of their servers? Is it a whale? Good on them for getting a statement out right away (although they basical…

> Is it a whale?

TikTok is my guess. ByteDance is valued at 250 billion. Plus, the change was pushed in the middle of the night, which would be daytime in Asia. Certainly there are other development teams in Asia, but considering the scale of the change it likely comes from HQ, and Fastly's whale in Asia would be them.

edit They may have lost TikTok at the end of last year, either partially or completely [1]. Anyone know what they use now? Akamai, or maybe they stealthily switched back to Fastly?

[1] https://www.barrons.com/articles/fastly-lost-tiktok-but-ceo-...

Re: Summary of June 8 outage

#29

Post by unityByFreedom »

johncolanduoni wrote: post #27
unityByFreedom wrote: post #23

Earlier quoted context omitted.

One of Cloudflare's top engineers previously wrote in this forum, > This incident emphasizes the importance of the Zero Trust model that Cloudflare follows and provides to customers, which ensures that if any one system or vendor is compromised, it does not compromise the entire organization. [1] Authentication is a part of a zero-trust model, not the whole thing. > No single specific technology is associated with ze…

They were referring to a completely different incident, involving compromised authentication to a camera system. I’d love to hear an explanation of how a zero-trust model would apply to this situation with Fastly. Seems like it would have to apply to a lot of multi-tenant resource exhaustion issues since we know so little about the specifics on the Fastly incident.

A blog post concerning how customer configurations cannot bring down other customers' sites would be great to see from Cloudflare. Fastly does not seem in a position to say that about its own stack and I don't expect another company to know their stack that well.

Re: Summary of June 8 outage

#30

Post by sersi »

EE84M3i wrote: post #21
plasma wrote: post #20

Earlier quoted context omitted.

At a guess, perhaps each server can serve everyone, and the system-wide config file that is shipped to each server (to handle each customer) perhaps became corrupted as part of a customer update, and it was rolled out to every node assumed to be correct. A defense against this could be to ensure the system that applies the change validates some health-checks continue to work after the new file is made (or automatical…

Yes, it sounds sort of like that. But this can be remediated by two things I was asking about: smoke tests and partitioning. When making a config change I'd assume they don't make it to all servers at once and instead roll it out gradually. If this caused the server to instantly start 503'ing all customers, presumably this would have been caught - perhaps it was more delayed though (resource leak, etc) and obviously…

One of the big competitive advantage Fastly has compared to say Akamai is that configuration changes roll out extremely fast. I could see them skimping on smoke tests to keep that advantage and not thinking that this could ever happen.
Post reply on HN