Live data from Hacker News

Summary of June 8 outage

fastly.com

51–60 of 110 posts

Re: Summary of June 8 outage

#51
post #38

This is annoyingly vague. What was the software bug, and what was the valid customer configuration change? It's perhaps a bit premature to demand it at this point, but I'm hoping a full post-mortem will outline precisely how this change was not picked up in pre-prod. Surely all valid customer configurations must be tested prior to rollout.

It's not just annoying value. It's insultingly vague. If my data centre provider suffered a complete outage, then I demand to get a detailed post-mortem of what happened (in due time). If they just tell me bullshit PR speak about "We value our customers", I'll be looking at switching providers. As a Fastly customer whose site went down, I'm entitled to know exactly what happened. If they don't tell me, I'm switching…

> As a Fastly customer whose site went down, I'm entitled to know exactly what happened.

Does your contract say you're entitled to an RCA?

As others have said, this is more of an update, not a complete RCA on the entire situation. They have short term tasks that they've described in this summary post and I would expect that they will give a more complete analysis later.

Re: Summary of June 8 outage

#52

Earlier quoted context omitted.

Welcome to the CentralizedWeb (tm). Unfortunately we had to sunset the Internet you used to use.

A web filled with DDOS attacks and scraping is a web that needs cloudflare and fastly. I’m not sure how to avoid this sorry state of things.

This attitude is why we have only 2½ search engines on the entire Internet. Only Google, Bing, and Yandex run crawlers. Everybody else is just a reseller for them.

Web crawlers are a feature not a bug. If your site shouldn't be crawled, it doesn't belong on the Internet.

Re: Summary of June 8 outage

#55

Earlier quoted context omitted.

Welcome to the CentralizedWeb (tm). Unfortunately we had to sunset the Internet you used to use.

A web filled with DDOS attacks and scraping is a web that needs cloudflare and fastly. I’m not sure how to avoid this sorry state of things.

Invisible Internet Project (I2P) is decentralized and defends from such attacks quite well.

Re: Summary of June 8 outage

#56

Earlier quoted context omitted.

A web filled with DDOS attacks and scraping is a web that needs cloudflare and fastly. I’m not sure how to avoid this sorry state of things.

This attitude is why we have only 2½ search engines on the entire Internet. Only Google, Bing, and Yandex run crawlers. Everybody else is just a reseller for them. Web crawlers are a feature not a bug. If your site shouldn't be crawled, it doesn't belong on the Internet.

Search engines scrapping your content is not the problem. Competitors scraping your content is.

Re: Summary of June 8 outage

#58
post #32

I love that somewhere out there is a developer who doesn't even work at Fastly but just innocently pushed a change to their Fastly config and basically broke the entire internet. I'm actually jealous. If it was me, I'd put that on my resume.

I can't actually imagine something like that can happen. Single person with a simple change in a config can cause this.

That was the bug.

A trivial example would be a bug that replaces the configuration for all customers with the last uploaded. Then when the next customer uploads a new (valid!) config, you have a problem.

Obviously it wasn’t that trivial but the point is: it wasn’t the customer’s configuration change that was the problem but some code that managed the config change.

Re: Summary of June 8 outage

#59
post #36

Earlier quoted context omitted.

I can't actually imagine something like that can happen. Single person with a simple change in a config can cause this.

They forgot to test it.

Test “it”? The change in question wasn’t by fastly but a customer of theirs making a config change. It’s possible that this customer did validate their change somehow.

Fastly obviously didn’t test their code (with the bug) enough, but testing of course can never prove the absence of bugs. Testing for a global deployment like a massive CDN happens to a large extent in prod because you don’t have another globe. You can test on a smaller scale but eventually you run into a problem that only shows itself at full scale.

Post reply on HN