Earlier quoted context omitted.
Does Cloudflare offer any hostname that also resolves to a different organization’s resolver (which must also have a TLS certificate for the Cloudflare hostname or DoH clients won’t be able to connect)?
Usually, for plain old DNS, primary and secondary resolvers are from the same provider, serving from distinct IPs.
Cloudflare 1.1.1.1 Incident on July 14, 2025
191–200 of 391 posts
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#192Earlier quoted context omitted.
Who? Honest question
Realistically, either you ignore the privacy concerns and set up routing to multiple providers preferring the fastest, or you go all-in on privacy and route DNS over Tor over bridge. Although, perhaps, having an external VPS with a dns proxy could be a good middle ground?
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#193This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#194This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1
If you were using systemd-resolved however, it retries all servers in the order they were specified, so it's important to interleave upstreams.
Using the servers in the above example, and assuming IPv4 + IPv6:
1.1.1.1
2001:4860:4860::8888
9.9.9.9
2606:4700:4700::1111
8.8.8.8
2620:fe::fe
1.0.0.1
2001:4860:4860::8844
149.112.112.112
2606:4700:4700::1001
8.8.4.4
2620:fe::9
will failover faster and more successfully on systemd-resolved, than if you specify all Cloudflare IPs together, then all Google IPs, etc.Also note that Quad9 is default filtering on this IP while the other two or not, so you could get intermittent differences in resolution behavior. If this is a problem, don't mix filtered and unfiltered resolvers. You definitely shouldn't mix DNSSEC validatng and not DNSSEC validating resolvers if you care about that (all of the above are DNSSEC validating).
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#195Interesting to see that they probably lost 20% of 1.1.1.1 usage from a roughly 20 minute incident. Not sure how cloudflare keeps struggling with issues like these, this isn't the first (and probably won't be the last) time they have these 'simple', 'deprecated', 'legacy' issues occuring. 8.8.8.8+8.8.4.4 hasn't had a global(1) second of downtime for almost a decade. 1: localized issues did exist, but that's really the…
There's more to DNS than just availability (granted, it's very important). There's also speed and privacy. European users might prefer one of the alternatives listed at https://european-alternatives.eu/category/public-dns over US corporations subject to the CLOUD act.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#196Earlier quoted context omitted.
Usually, for plain old DNS, primary and secondary resolvers are from the same provider, serving from distinct IPs.
Yes, but you were talking about DoH. I don’t know how that could plausibly work.
DoH hosts can resolve to multiple IPs (and even different IPs for different clients)?
Also see TFA
It's worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. DoH remained available and traffic was mostly unaffected as cloudflare-dns.com uses a different set of IP addresses.Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#197Question: Years ago, back when I used to do networking, Cisco Wireless controllers used 1.1.1.1 internally. They seemed to literally blackhole any comms to that IP in my testing. I assume they changed this when 1.0.0.0/8 started routing on the Internet?
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#198Earlier quoted context omitted.
There's a constant tension between speed of detection and false positive rates. Traditional monitoring systems like Nagios and Icinga have settings where they only open events/alerts if a check failed three times in a row, because spurious failed checks are quite common. If you spam your operators with lots of alerts for monitoring checks that fix themselves, you stress the unnecessarily and create alert blindness, b…
I work on the SSO stack in a b2b company with about 200k monthly active users. One blind spot in our monitoring is when an error occurs on the client's identity provider because of a problem on our side. The service is unusable and we don't have any error logs to raise an alert. We tried to setup an alert based on expected vs actual traffic but we concluded that it would create more problems for the reason you provid…
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#199Earlier quoted context omitted.
At Cloudflare’s scale on 1.1.1.1, I’d imagine you could do something comparatively simple like track ten-minute and ten-second rolling averages (I know, I know, I make that sound much easier and more practical than it actually would be), and if they differ by more than 50%, sound the alarm. (Maybe the exact numbers would need to be tweaked, e.g. 20 seconds or 80%, but it’s the idea.) Were it much less than 1.1.1.1 it…
I'm sure some engineer at cloudflare is evaluating something like this right now, and try it on historical data how many false positives that would've generated in the past, if any. Thing is, it's probably still some engineering effort, and most orgs only really improve their monitoring after it turned out to be sub-optimal.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#200 Cloudflare's 1.1.1.1 Resolver service became unavailable to the Internet starting at 21:52 UTC and ending at 22:54 UTC
Weird. According to my own telemetry from multiple networks they were unavailable for a lot longer than that.