Earlier quoted context omitted.
Middle of the night? It was at noon for most of Europe
What are Europe's whales?
gov.uk ?
Summary of June 8 outage
81–90 of 110 posts
Re: Summary of June 8 outage
#82You took down my site and a good swath of the whole internet. I am entitled to know, in detail, what happened, so I can be more informed and assess any actions I might need to take. I don't want any more of your PR speak or "we value our customers". That's crap and insults my intelligence. STOP getting PR to write your comms; just speak to engineers like engineers. I'd rather get no response than this post. I hope th…
There's no flawless system; all you need to know really is that it'll have some downtime sometimes. If this is unacceptable to you...
> any actions I might need to take
An alternative CDN setup you can switch to when there are problems.
Re: Summary of June 8 outage
#83I might sound naive, but as a true hyperscale internet company to plan for disaster scenario like this as a consumer of fastly? How could you plan for an outage like this by fastly and how could you mitigate this?
Use a short TTL on the CDN subdomain you use. Then setup an alternative CDN provider in advance, so that you can switch from one to the other in a matter of minutes.
Using a short TTL on a DNS entry has performance issues [1]. But a long one also isn't ideal [2].
[1] https://twitter.com/RichardTowers/status/1402312111078903812
[2] https://twitter.com/RichardTowers/status/1402312308077076484
Re: Summary of June 8 outage
#84Earlier quoted context omitted.
Invisible Internet Project (I2P) is decentralized and defends from such attacks quite well.
How is that project doing? It's been around for years and does not come up often.
Not sure why it's not adopted more widely. It's in constant development and is impressive.
Re: Summary of June 8 outage
#85Earlier quoted context omitted.
Are you implying that no senior devs or main offices are located in europe? Also, just because the rollout to fastly happened at a morning EU time, doesn't mean that the change was made. If there's a deployment pipeline, it could have been made 2-3 hours earlier, or even the day before
I'm saying a big network change is probably going to be executed while HQ is awake and in the loop. In my experience the most skillful devs aren't the most senior.
We don't know much about the specific client configuration change that triggered this condition yet. It doesn't necessarily have to be a big company wide infrastructure change.
To me it sounds plausible that an SRE team in an alternate location made a change scoped to their permission level, following company-directed playbooks, which eventually triggered the faulty condition at Fastly.
Re: Summary of June 8 outage
#86I love that somewhere out there is a developer who doesn't even work at Fastly but just innocently pushed a change to their Fastly config and basically broke the entire internet. I'm actually jealous. If it was me, I'd put that on my resume.
I can't actually imagine something like that can happen. Single person with a simple change in a config can cause this.
It's more common than we imagine. That's usually the start of many of the historical network incidents. The important part, as usual, is to make sure the remediations of such incidents focus on how to limit blast radius of small changes, and how to accomplish that without imposing artificial gatekeeping and bureaucracy into the change process.
Re: Summary of June 8 outage
#87I love that somewhere out there is a developer who doesn't even work at Fastly but just innocently pushed a change to their Fastly config and basically broke the entire internet. I'm actually jealous. If it was me, I'd put that on my resume.
Really weird case of infinite number of monkeys. I wonder if fuzz testing would have found this one.
Re: Summary of June 8 outage
#88Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration change safety mechanisms/smoke tests/gradual rollout. Also, what is up with their partitioning? Do they seriously have one customer that gets served from 85% of their servers? Is it a whale? Good on them for getting a statement out right away (although they basical…
At its core Fastly uses heavily modified Varnish 2 with customer's having full access to vcl providing an unprecedented level of control of the request processing and cache behavior. It is extremely difficult, if not impossible, to completely remove foot-guns while retaining this ability. They do amazing thing checking validity of the code and aborting broken requests but it is still vcl and in its core varnish 2 doe…
Thank you for that. That last clause seems like the real key here. With that much power in customers' hands and no hard multi-tenant isolation, unintentional DoS becomes almost inevitable. In effect, it puts every customer at the mercy of every other's diligence (or lack thereof). Even with diligence, that seems a bit fragile.
Re: Summary of June 8 outage
#89Earlier quoted context omitted.
Search engines scrapping your content is not the problem. Competitors scraping your content is.
If you don't want your content crawled, don't put it on the public Internet. Your profitability is not the Internet's problem.
It’s a very interesting proposition.
If you cannot generate revenue by your internet content, probably you can’t live from generating content for the internet.
The consequence, IMHO, is that the internet would have this amount of content and usefulness.
Newspapers? No. Can’t live from internet news if anyone can copy a reporter’s work and post it on his own site and dilute traffic.
Online selling? Don’t look like a viable business model, as anyone can copy the photos you paid a photographer for, the descriptions you paid someone to write and the reviews your customers wrote. True reviews are priceless, you now? Even more now that an AI can detect computer generated reviews.
Obviously an open and totally money-free internet is nice, but it wouldn’t be the internet people make a living from.
Re: Summary of June 8 outage
#90This is annoyingly vague. What was the software bug, and what was the valid customer configuration change? It's perhaps a bit premature to demand it at this point, but I'm hoping a full post-mortem will outline precisely how this change was not picked up in pre-prod. Surely all valid customer configurations must be tested prior to rollout.
It's not just annoying value. It's insultingly vague. If my data centre provider suffered a complete outage, then I demand to get a detailed post-mortem of what happened (in due time). If they just tell me bullshit PR speak about "We value our customers", I'll be looking at switching providers. As a Fastly customer whose site went down, I'm entitled to know exactly what happened. If they don't tell me, I'm switching…
> As a Fastly customer whose site went down, I'm entitled to know exactly what happened. If they don't tell me, I'm switching CDNs as a matter of priority.|
If you are a hardcore user of their vcl on the edge I'm very curious where you would go to. The last time I looked ( a year ago ) there was no one that came even close to giving customers that level of control in request processing. Most of them fail do complicated stuff with CORSs without doing arabesque while balancing on a medicine ball ( Looking at you Lambda@Edge ) not to mention ability to massage the response.