Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Summary of June 8 outage

fastly.com

Post reply on HN
Print view View original

11–20 of 110 posts

Re: Summary of June 8 outage

#11

Post by unityByFreedom »

fafner wrote: post #8
unityByFreedom wrote: post #2

So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-i...

Zero trust seems to be very unrelated to this issue. The issue seems to have been a poison config breaking fastly stack. Zero trust is about verifying authentication of devices/users. Unrelated things really.

It's much more than that. It's a whole approach to architecture. The post I linked says it best.

Re: Summary of June 8 outage

#14

Post by fafner »

unityByFreedom wrote: post #11
fafner wrote: post #8

Earlier quoted context omitted.

Zero trust seems to be very unrelated to this issue. The issue seems to have been a poison config breaking fastly stack. Zero trust is about verifying authentication of devices/users. Unrelated things really.

It's much more than that. It's a whole approach to architecture. The post I linked says it best.

Authentication/trust is simply unrelated to this outage. The post you linked seems to have no relevance.

Re: Summary of June 8 outage

#15

Post by thethethethe »

unityByFreedom wrote: post #2

So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-i...

> commitment to ensure that customer configurations cannot break the whole system

You can't just ensure a config change won't break things in large distributed systems, it's too complex with too many factors, there will always be risk. To mitigate your risk, youd want to design your system to do progressive, regional rollouts, with canaries to attempt to detect and isolate before a wide spread outage occurs. Even if you have all of this set up, there is still risk that your regions and systems are not fully isolated and outages could cascade anyway.

There will always be risk, there will always be errors. This is why SLAs and SLOs exist, they define and codify an agreement of what an outage is and what compensation is required if the agreement isn't met.

You can read Fastlys SLA here: https://docs.fastly.com/products/service-availability-sla

Re: Summary of June 8 outage

#17

Post by abestic9 »

EE84M3i wrote: post #7

Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration change safety mechanisms/smoke tests/gradual rollout. Also, what is up with their partitioning? Do they seriously have one customer that gets served from 85% of their servers? Is it a whale? Good on them for getting a statement out right away (although they basical…

It's an update on the situation until they confirm the bug fix is completely rolled out. I certainly wouldn't expect they tell their customer base how they can exploit a bug to bring down part of their business.

With respect to partitioning - we don't know how or why an invalid configuration could poison so many nodes; if the config was physically present on them or if there was a cascade of healing/balancing issues stemming from it.

I would leave speculation on many of your points at the doorstep until we see a full report.

Re: Summary of June 8 outage

#19

Post by fafner »

thethethethe wrote: post #15
unityByFreedom wrote: post #2

So, a valid customer configuration change triggered a bug. One thing I don't see in this writeup is a commitment to ensure that customer configurations cannot break the whole system. Cloudflare does seem to make this promise with their zero trust architecture, https://www.cloudflare.com/learning/security/glossary/what-i...

> commitment to ensure that customer configurations cannot break the whole system You can't just ensure a config change won't break things in large distributed systems, it's too complex with too many factors, there will always be risk. To mitigate your risk, youd want to design your system to do progressive, regional rollouts, with canaries to attempt to detect and isolate before a wide spread outage occurs. Even if…

Unfortunately some customer changes will need to go out quickly and globally for a CDN. Going much slower might not be a good option.

Canarying should detect this. Not clear if they do this or the canary failed to report this.

Sharding by customers could help reduce blast radius. But maybe not by much of this was a very big customer.

Re: Summary of June 8 outage

#20

Post by plasma »

EE84M3i wrote: post #7

Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration change safety mechanisms/smoke tests/gradual rollout. Also, what is up with their partitioning? Do they seriously have one customer that gets served from 85% of their servers? Is it a whale? Good on them for getting a statement out right away (although they basical…

At a guess, perhaps each server can serve everyone, and the system-wide config file that is shipped to each server (to handle each customer) perhaps became corrupted as part of a customer update, and it was rolled out to every node assumed to be correct.

A defense against this could be to ensure the system that applies the change validates some health-checks continue to work after the new file is made (or automatically rollback to previous configuration).

I can see how this would happen, assuming thats what happened.

Post reply on HN