Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare Global Network experiencing issues

cloudflarestatus.com

Post reply on HN
Print view View original

981–990 of 1001 posts

Re: Cloudflare Global Network experiencing issues

#981

Post by immibis »

__turbobrew__ wrote: post #950
aavshr wrote: post #939

> In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. From the CTO, Source: https://x.com/dok2001/status/1990791419653484646

It still astounds me that the big dogs still do not phase config rollouts. Code is data, configs are data, they are one and the same. It was the same issue with the giant crowdstrike outage last year, they were rawdogging configs globally and a bad config made it out there and everything went kaboom. You NEED to phase config rollouts like you phase code rollouts.

Because adversaries adapt quickly, they have a system that deploys their counter-adversary bits quickly without phasing - no matter whether they call them code or configs. See also: Crowdstrike.

Re: Cloudflare Global Network experiencing issues

#983

Post by RKFADU_UOFCCLEL »

JoeOfTexas wrote: post #978
abelanger wrote: post #783

If anyone needs commands for turning off the CF proxy for their domains and happens to have a Cloudflare API token. First you can grab the zone ID via: curl -X GET "https://api.cloudflare.com/client/v4/zones" -H "Authorization: Bearer $API_TOKEN" -H "Content-Type: application/json" | jq -r '.result[] | "\(.id) \(.name)"' And a list of DNS records using: curl -X GET "https://api.cloudflare.com/client/v4/zones/$ZONE_ID…

I would advise against this action. Just ride the crash.

If people knew how to play the 5 hour long game they wouldn't have been using Cloudflare in the first place.

Re: Cloudflare Global Network experiencing issues

#985

Post by whalesalad »

rco8786 wrote: post #503

Is it me or has there been a very noticeable uptick in large scale infra-level outages lately? AWS, Cloudflare, etc have all been way under whatever SLA they publish.

Somewhere, at a floating desk behind a wall of lava lamps, in a nyancatified ghostty terminal with 32 different shader plugins installed:

You're absolutely right! I shouldn't have force pushed that change to master. Let me try and roll it back. * Confrobulating* Oh no! Cloudflare appears to be down and I cannot revert the change. Why don't you go make a cup of coffee until that comes back. This code is production ready, it's probably just a blip.

Re: Cloudflare Global Network experiencing issues

#986

Post by hoistbypetard »

Freak_NL wrote: post #28
itzjacki wrote: post #10

A colleague of mine just came bursting through my office door in a panic, thinking he brought our site down since this happened just as he made some changes to our Cloudflare config. He was pretty relieved to see this post.

Is there a word for that feeling of relief when someone else fucked up after initially thinking it was you?

schadenfuckup

Re: Cloudflare Global Network experiencing issues

#987

Post by himinlomax »

__turbobrew__ wrote: post #950
aavshr wrote: post #939

> In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. From the CTO, Source: https://x.com/dok2001/status/1990791419653484646

It still astounds me that the big dogs still do not phase config rollouts. Code is data, configs are data, they are one and the same. It was the same issue with the giant crowdstrike outage last year, they were rawdogging configs globally and a bad config made it out there and everything went kaboom. You NEED to phase config rollouts like you phase code rollouts.

You can't protect against _latent bugs_ with phased rollouts.

Re: Cloudflare Global Network experiencing issues

#988

Post by kazen44 »

omani wrote: post #976

one way to mitigate DDoS is to enforce source IP checks on the way OUT of a datacenter (egress). sure there are botnets, infected devices, etc that would conform to this but where does the sheer power of a big ddos attack come from? including those who sell it as a service. they have to have some infrastructure in some datacenter right? make a law that forces every edge router of a datacenter to check for source IP a…

alright, what you are proposing is kind of hard to do. Source routing is not easy, and source validations is even harder.

and what prevents me, as a abuse hoster or "bad guy" from just announcing my own IP space directly on a transit or IXP?

You might say, the IXP should do source checking aswell, but what if ipspace is distributed/anycasted across multiple ASN's/ on the IXP?

Also, if you add multiple egress points distributed across different routing domains, it gets complicated fast.

Does my transit upstream need to do source validation of my IP space? What about their upstream? Also, how would he know which IPspace belongs to which ASN's considering the allocation of ASN numbers and IP space is distributed across different organisations across the globe. (some of which are more malicious/non function than others[0]). Source routing becomes extremly complex because there is no single, universal mapping between IP space and ASN's they belong too.

[0]https://afrinic.net/notice-for-termination-of-the-receiversh...

Post reply on HN