Honestly Cloudflare's PR pissed me off yesterday. Straight up going on LinkedIn and other socials telling everything was solved in one hour (actually 37 minutes), even though I and many other companies I know still had issues with their services *16 hours after* the post. Those are things that make me reconsider my position with Cloudflare. Straight up lying and not verifying whether your customers are able to operat…
Cloudflare outage – 24 hours now
71–76 of 76 posts
Re: Cloudflare outage – 24 hours now
#72Earlier quoted context omitted.
I think you might have missed the joke on this one.
Honestly, the original post could have been a joke, or it could not have been. I regularly talk to people who seem to genuinely believe this sort of thing (on this topic and others).
In that era we also saw the last sysadmin configuring Apache with their bare hands without the help of Cloudflare.
Re: Cloudflare outage – 24 hours now
#73Looking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event. I understand we do not have the technology for that just yet, and DevOps able to configure TLS terminators on their own are worth their weight in gold. Hard to imagine how the Internet could ever exist without Cloudflare.
Missing the /s I hope.
Re: Cloudflare outage – 24 hours now
#74I've heard tell of massive DDoS attacks against international news sources (AP, Reuters, NY Times). Not sure if this is related.
In this case it is not. A power outage in a critical data-center is the root cause here: https://www.cloudflarestatus.com/incidents/hm7491k53ppg
Re: Cloudflare outage – 24 hours now
#75https://blog.cloudflare.com/post-mortem-on-cloudflare-contro...
"On November 2 at 08:50 UTC Portland General Electric (PGE), the utility company that services PDX-04, had an unplanned maintenance event affecting one of their independent power feeds into the building. That event shut down one feed into PDX-04. The data center has multiple feeds with some level of independence that can power the facility. However, Flexential powered up their generators to effectively supplement the feed that was down.
Counter to best practices, Flexential did not inform Cloudflare that they had failed over to generator power. None of our observability tools were able to detect that the source of power had changed. Had they informed us, we would have stood up a team to monitor the facility closely and move control plane services that were dependent on that facility out while it was degraded.
It is also unusual that Flexential ran both the one remaining utility feed and the generators at the same time. It is not unusual for utilities to ask data centers to drop off the grid when power demands are high and run exclusively on generators. Flexential operates 10 generators, inclusive of redundant units, capable of supporting the facility at full load. It would also have been possible for Flexential to run the facility only from the remaining utility feed. We haven't gotten a clear answer why they ran utility power and generator power."
Re: Cloudflare outage – 24 hours now
#76Looking forward to a more decentralised global Internet, with packets being routed through alternative paths, so outages like these become a non-event. I understand we do not have the technology for that just yet, and DevOps able to configure TLS terminators on their own are worth their weight in gold. Hard to imagine how the Internet could ever exist without Cloudflare.