91–100 of 110 posts
Thought it surprising that amazon.com uses fastly when they have their own cdn
Someone answered this yesterday. CloudFront is good for video and large download assets (plus very low margins) but not for images and smaller stuff which Fastly is much faster at: https://www.streamingmediablog.com/2020/05/fastly-amazon-hom...
Earlier quoted context omitted.
I don't know about you but I find the prospect of a Cloudflare monoculture pretty worrying, especially since they've already demonstrate a willingness to kick off users they don't like. (I also think the https veneer that they offer is misleading to end users and bad for everyone on the internet, though not everyone will agree with that).
I too think Cloudflare's "reverse HTTPS proxy" approach (where they have a CA-signed certificate for every domain that gets pointed at them) is bad for the Internet. But how does Fastly avoid this problem? It's really more a symptom of the "web pki" trainwreck than anything else. I tried looking on Fastly's website for technical details, but like every other corporate website it was an impenetrable mass of marketing…
Earlier quoted context omitted.
I'm saying a big network change is probably going to be executed while HQ is awake and in the loop. In my experience the most skillful devs aren't the most senior.
We don't know much about the specific client configuration change that triggered this condition yet. It doesn't necessarily have to be a big company wide infrastructure change. To me it sounds plausible that an SRE team in an alternate location made a change scoped to their permission level, following company-directed playbooks, which eventually triggered the faulty condition at Fastly.
I might sound naive, but as a true hyperscale internet company to plan for disaster scenario like this as a consumer of fastly? How could you plan for an outage like this by fastly and how could you mitigate this?
I was thinking more about this though and it has its own problems. You want a short TTL so failover is fast, but this increases the number of DNS lookups people have to do (and DNS lookups can be very slow!).
Additionally, a short TTL means you're more vulnerable to problems like the dyndns attack [1] from 2016: names with longer TTLs were up for longer since they preserved the correct DNS records for longer.
But if you have a long TTL, even if you fail over, you'll still be down for at least as long as the DNS TTL pointing to the bad CDN.
Maybe, you could do DNS roundrobin against multiple CDN providers at once. Say you used 4, then if one went down, only 25% of requests would fail, and you could just remove the failing entry. This seems very expensive!
Honestly, the cost of these solutions is probably not worth it. The product I work on went partially down during the fastly outage. Then it came back up and everything is back to normal. It really won't impact us much at all. Shrug.
Absolutely no details about the bug or why a single customer configuration effected global state on the server, or why this wasn't caught by configuration change safety mechanisms/smoke tests/gradual rollout. Also, what is up with their partitioning? Do they seriously have one customer that gets served from 85% of their servers? Is it a whale? Good on them for getting a statement out right away (although they basical…
I don't expect fastly to name and shame a customer who made a valid change, nor do I expect fastly to give us a detailed explanation of what the bug is. I'm still a little annoyed at their status page [0]. It says: > We're currently investigating potential impact to performance with our CDN services. yet in the blog post we're talking about here it says: > Early June 8, a customer pushed a valid configuration change…
They say it was a performance issue, but we were getting 500 errors from the Fastly API.
My guess: it is some sort of config-triggered recursion that caused the servers to stack overflow and crash/reboot in a cycle.
Earlier quoted context omitted.
How is that project doing? It's been around for years and does not come up often.
Not sure why it's not adopted more widely. It's in constant development and is impressive.
Can it be used as a CDN for a normal website? How well does it perform?