What's ironic is that my experience of Cloudflare Sales is that they take advantage of any downtime from their competitor to try and get people to migrate to their services...
Beyond scummy
Cloudflare Network Performance Issues
321–329 of 329 posts
Re: Cloudflare Network Performance Issues
#322Once cloudflare.com came back I decided to check out their business SLA, and it's not very encouraging: > For any and each Outage Period during a monthly billing period the Company will provide as a Service Credit an amount calculated as follows: Service Credit = (Outage Period minutes * Affected Customer Ratio) ÷ Scheduled Availability minutes - https://www.cloudflare.com/business-sla/ So assuming an outage affects…
I left Cloudflare for AWS a long time ago despite CF's affordability since they didn't seem to care that much about uptime or quality. Their frontend was corrupting response bodies + caching response bodies (in retrospect, this was probably pre-discovery cloudbleed) and there was no way to get a response or help with it.
Re: Cloudflare Network Performance Issues
#323Earlier quoted context omitted.
Sadly this reminds me of AWS outages too where the same applies. How is it that hundreds of developers know there's an issue before AWS do, or Cloudflare in this instance. See my blog post on similar AWS uptime reporting issues at https://www.ably.io/blog/honest-status-reporting-aws-service . At Ably, our status site had an incident update about Cloudflare issues being worked on (by routing away from CF) before Cloud…
At least in the case with AWS, unfortunately there's business involved - because of their uptime guarantee, incidents that would be called downtime by a purely technical team are left as "operational" or "partly degraded". Otherwise, they might have to shell out millions or tens of millions.
You have to provide "your request logs that document the errors and corroborate your claimed outage" for the AWS Compute SLA https://aws.amazon.com/compute/sla/
Re: Cloudflare Network Performance Issues
#324They have now released an initial statement [1]: For about 30 minutes today, visitors to Cloudflare sites received 502 errors caused by a massive spike in CPU utilization on our network. This CPU spike was caused by a bad software deploy that was rolled back. Once rolled back the service returned to normal operation and all domains using Cloudflare returned to normal traffic levels. This was not an attack (as some ha…
> Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide.
Sounds like backtracking. If so, I'll bet there's a conversation happening about switching to re2.
edit: hmmm, https://github.com/cloudflare/lua-re2
I'm also curious why this rollout wasn't staged.
Re: Cloudflare Network Performance Issues
#325It's what we get for centralizing this here internet, folks.
Re: Cloudflare Network Performance Issues
#326Earlier quoted context omitted.
Heh, I am reminded of when the control plane at AWS went down... and we had a custom autoscaling config that would query for the number of instances running and scale appropriately... but when the AWS API died... we kept getting zero running instances... So our system thought none were running and so it kept launching instances.... These were SPOT instances and thus only cost like .10 per hour... But we launched like…
> but he created the 201th repo (we had only paid for 200) That's an odd choice of a failure mode. > AWS still refunded all costs... Yeah they should. It was their silly design choice that lead to disclosure of secrets after all. What kind of failure mode is that even. Failing to create the repo would have led to a better user experience for sure. Can you imagine if S3 charged more for private objects and once you re…
>It was their silly design choice that lead to disclosure of secrets after all.
Wait, was the 200 private repos issue an AWS thing or a GitHub/GitLab/whatever thing?
What AWS product has a concept of private/public repos and limits on how many of the former you can get for a certain price?
Re: Cloudflare Network Performance Issues
#327Earlier quoted context omitted.
> but he created the 201th repo (we had only paid for 200) That's an odd choice of a failure mode. > AWS still refunded all costs... Yeah they should. It was their silly design choice that lead to disclosure of secrets after all. What kind of failure mode is that even. Failing to create the repo would have led to a better user experience for sure. Can you imagine if S3 charged more for private objects and once you re…
>It was their silly design choice that lead to disclosure of secrets after all. Wait, was the 200 private repos issue an AWS thing or a GitHub/GitLab/whatever thing? What AWS product has a concept of private/public repos and limits on how many of the former you can get for a certain price?
It was a git thing.
Never post aws secrets to git.
Re: Cloudflare Network Performance Issues
#328"Unfortunately, one of these rules contained a regular expression that caused CPU to spike to 100% on our machines worldwide. This 100% CPU spike caused the 502 errors that our customers saw."
So, it was a dirty REGEX. I can't even be mad.
Re: Cloudflare Network Performance Issues
#329The Cloudflare DNS-over-HTTPS resolver was serving up 502 errors as well, though the standard port 53 UDP resolver was working. This event definitely made me regret choosing Cloudflare as my sole DoH server.
Hear hear Mozilla!