71–80 of 391 posts
Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.
I’m surprised at the delay in impact detection: it took their internal health service more than five minutes to notice (or at least alert) that their main protocol’s traffic had abruptly dropped to around 10% of expected and was staying there. Without ever having been involved in monitoring at that kind of scale, I’d have pictured alarms firing for something that extreme within a minute. I’m curious for description o…
There's a constant tension between speed of detection and false positive rates. Traditional monitoring systems like Nagios and Icinga have settings where they only open events/alerts if a check failed three times in a row, because spurious failed checks are quite common. If you spam your operators with lots of alerts for monitoring checks that fix themselves, you stress the unnecessarily and create alert blindness, b…
Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed…
I disagree. The actual root cause here is shrouded in jargon that even experienced admins such as myself have to struggle to parse. It’s corporate newspeak. “legacy” isn’t a clear term, it’s used to abstract and obfuscate. > Legacy components do not leverage a gradual, staged deployment methodology. Cloudflare will deprecate these systems which enables modern progressive and health mediated deployment processes to pr…
I will not say whether or not it’s acceptable for a company of their size and maturity, but it’s definitely not hidden in corporate lingo.
I do believe they could have elaborate more on the follow up steps they will take to prevent this from happening again, I don’t think staggered roll outs are the only answer to this, they’re just a safety net.
It's crazy that both 1.1.1.1 and 1.0.0.1 where affected by the same change I guess now we should start using a completely different provider as dns backup Maybe 8.8.8.8 or 9.9.9.9
In general, the idea of DNS's design is to use the DNS resolver closest to you, rather than the one run by the largest company. That said, it's a good idea to specifically pick multiple resolvers in different regions, on different backbones, using different providers, and not use an Anycast address, because Anycast can get a little weird. However, this can lead to hard-to-troubleshoot issues, because DNS doesn't alwa…
Earlier quoted context omitted.
How does DoH work? Somehow you need to know the IP of cloudflare-dns.com first. Maybe your router uses 1.1.1.1 for this.
Yeah I don’t understand this part either, maybe it’s supposed to be bootstrapped using your ISP’s DNS server?
> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.
Earlier quoted context omitted.
1.1.1.1 and 1.0.0.1 are served by the same service. It's not advertised as a redundant fully separate backup or anything like that...
Wait, then why does 1.0.0.1 exist? I'll grant I've never seen it advertised/documented as a backup, but I just assumed it must be because why else would you have two? (Given that 1.1.1.1 already isn't actually a single point, so I wouldn't think you need a second IP for load balancing reasons.)