951–960 of 1001 posts
> In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. From the CTO, Source: https://x.com/dok2001/status/1990791419653484646
> In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. From the CTO, Source: https://x.com/dok2001/status/1990791419653484646
It still astounds me that the big dogs still do not phase config rollouts. Code is data, configs are data, they are one and the same. It was the same issue with the giant crowdstrike outage last year, they were rawdogging configs globally and a bad config made it out there and everything went kaboom. You NEED to phase config rollouts like you phase code rollouts.
How did we get to a place where either Cloudflare or AWS having an outage means a large part of the web going down? This centralization is very worrying.
How did we get to a place where either Cloudflare or AWS having an outage means a large part of the web going down? This centralization is very worrying.
How did we get to a place where either Cloudflare or AWS having an outage means a large part of the web going down? This centralization is very worrying.
How did we get to a place where either Cloudflare or AWS having an outage means a large part of the web going down? This centralization is very worrying.
And it’s hard to protect against DDoS without something like Cloudflare.
Look at the posts here.
Even the meager HN “hug of death” will take things down
> In short, a latent bug in a service underpinning our bot mitigation capability started to crash after a routine configuration change we made. That cascaded into a broad degradation to our network and other services. This was not an attack. From the CTO, Source: https://x.com/dok2001/status/1990791419653484646
It still astounds me that the big dogs still do not phase config rollouts. Code is data, configs are data, they are one and the same. It was the same issue with the giant crowdstrike outage last year, they were rawdogging configs globally and a bad config made it out there and everything went kaboom. You NEED to phase config rollouts like you phase code rollouts.
I must say I'm astonished, as naive as it may be, to see the number of separate platforms affected by this. And it has been a bit of a learning experience too.