41–50 of 211 posts
I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.
for example cloudflare tunnels will not work if restarted, our production is running with 1 single tunnel now.
Bad timing: quarterly earnings call is this afternoon.
I dunno. Cloudflare gives me the creeps. I have no idea why so many folks think large swaths of the Internet should be reliant on a single company.
No free (or even cheap) alternatives exist. If you have a little site that might be a DoS target, you have to use it.
It seems to be a common misconception that people defaults to, that you have to use CloudFlare or some alternative, otherwise you'll get hacked/ddos'd for sure.
The lifecycle is: PoPs generate/gather data > send to PDX > compute in PDX > ship updates / data to PoPs.
If you take out PDX, then as so much runs on fresh data, it starts getting stale.
I doubt everything has changed since then, so this is unlikely just "API down" and more likely that a lot of things are now in a degraded state as they're running on stale information (no update from PDX)... this includes things like load balancing, the tiered caching (Argo Smart Routing), Warp / Zero Trust, etc.
Even if it were only "API down", then bear in mind that a lot of automation customers have will block attacks by calling the API... "API down" is a hell of a window of opportunity for attackers.
Note that just before I'd left they'd been investing in standing up AMS (I think) but had never successfully tested a significant failover, and the majority of services that needed fresh state did not know how to do this.
PS: :scream: most of the observability was also based in PDX, so hugs to all the teams and SREs currently running blind.
Earlier quoted context omitted.
If only there was a protocol of some sorts that allowed you to send/receive code and patches even if the centralized hub everyone uses for synchronization was down...
Repeat after me: GitHub is more than just Git. GitHub is more than just Git.
This is not looking good. I really hope they don't move away from dogfooding because of this. Also vercel deployments (and edge functions in general) aren't working. Their status page says: "We identified the root cause as an issue with one of our upstream providers and working with them towards mitigation." I wonder if edge functions use workers under the hood lol. LINK: https://www.vercel-status.com/
> Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services.
Someone else in the thread shared this thread [0] which is about Flexential PDX02 losing power at the same time.
> Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site.
Sounds like the entire DC went down and their failover didn't handle it as gracefully as it should have.
[0]: https://puck.nether.net/pipermail/outages/2023-November/0149...
Earlier quoted context omitted.
Repeat after me: GitHub is more than just Git. GitHub is more than just Git.
I would like to see Git be extended with a decentralized approach to bugtracking, code reviews, wiki's for documentation etc.
https://fossil-scm.org/home SCM, bug tracker, wiki...
When I worked there (3+ years ago), if PDX were out then "the brain" was out... things like DDoS protection was already being done within each PoP (so that will be just fine, even for L3 and L7 floods, even for new and novel attacks), but nearly everything else was done with the compute in PDX and then shipped to each PoP as configuration data. The lifecycle is: PoPs generate/gather data > send to PDX > compute in PD…
> Cloudflare is assessing a loss of power impacting data centres while simultaneously failing over services.
> [0]: Looks like they lost utility, switched to generator, and then generator failed (not clear on scope of Gen failure yet). Some utility power is back, so recovery is in progress for some portion of the site.
[0]: https://puck.nether.net/pipermail/outages/2023-November/0149...