Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

191–200 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#191
post #183

Earlier quoted context omitted.

Does Cloudflare offer any hostname that also resolves to a different organization’s resolver (which must also have a TLS certificate for the Cloudflare hostname or DoH clients won’t be able to connect)?

Usually, for plain old DNS, primary and secondary resolvers are from the same provider, serving from distinct IPs.

Yes, but you were talking about DoH. I don’t know how that could plausibly work.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#192
post #41

Earlier quoted context omitted.

Who? Honest question

Realistically, either you ignore the privacy concerns and set up routing to multiple providers preferring the fastest, or you go all-in on privacy and route DNS over Tor over bridge. Although, perhaps, having an external VPS with a dns proxy could be a good middle ground?

If you're the technical type you can run Unbound locally (even on Windows) and let it forward queries with DoT. No need for neither Tor nor running your own external resolver.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#193

This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1

Even without "all-servers", DNSMasq will race servers frequently (after 20 seconds, unless it's changed), and when retrying. A sudden outage should only affect you for a few seconds, if at all.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#194

This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1

Additionally, as long as you don't set strict-order, dnsmasq will automatically use all-servers for retries.

If you were using systemd-resolved however, it retries all servers in the order they were specified, so it's important to interleave upstreams.

Using the servers in the above example, and assuming IPv4 + IPv6:

    1.1.1.1
    2001:4860:4860::8888
    9.9.9.9
    2606:4700:4700::1111
    8.8.8.8
    2620:fe::fe
    1.0.0.1
    2001:4860:4860::8844
    149.112.112.112
    2606:4700:4700::1001
    8.8.4.4
    2620:fe::9
will failover faster and more successfully on systemd-resolved, than if you specify all Cloudflare IPs together, then all Google IPs, etc.

Also note that Quad9 is default filtering on this IP while the other two or not, so you could get intermittent differences in resolution behavior. If this is a problem, don't mix filtered and unfiltered resolvers. You definitely shouldn't mix DNSSEC validatng and not DNSSEC validating resolvers if you care about that (all of the above are DNSSEC validating).

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#195
post #127

Interesting to see that they probably lost 20% of 1.1.1.1 usage from a roughly 20 minute incident. Not sure how cloudflare keeps struggling with issues like these, this isn't the first (and probably won't be the last) time they have these 'simple', 'deprecated', 'legacy' issues occuring. 8.8.8.8+8.8.4.4 hasn't had a global(1) second of downtime for almost a decade. 1: localized issues did exist, but that's really the…

There's more to DNS than just availability (granted, it's very important). There's also speed and privacy. European users might prefer one of the alternatives listed at https://european-alternatives.eu/category/public-dns over US corporations subject to the CLOUD act.

I think just setting up Unbound is even less trouble. Servers come and go. Getting rid of the dependency altogether is better than having to worry who operates the DNS-servers and how long it's going to be available.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#196
post #191

Earlier quoted context omitted.

Usually, for plain old DNS, primary and secondary resolvers are from the same provider, serving from distinct IPs.

Yes, but you were talking about DoH. I don’t know how that could plausibly work.

> but you were talking about DoH

DoH hosts can resolve to multiple IPs (and even different IPs for different clients)?

Also see TFA

  It's worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. DoH remained available and traffic was mostly unaffected as cloudflare-dns.com uses a different set of IP addresses.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#197

Question: Years ago, back when I used to do networking, Cisco Wireless controllers used 1.1.1.1 internally. They seemed to literally blackhole any comms to that IP in my testing. I assume they changed this when 1.0.0.0/8 started routing on the Internet?

The general guidance for networking has been to only use IPs and domains that you actually control... But even 5-8 years ago, the last time I personally touched a cisco WLC box, it still had 1.1.1.1 hardcoded. Cisco loves to break their own rules...

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#198
post #72

Earlier quoted context omitted.

There's a constant tension between speed of detection and false positive rates. Traditional monitoring systems like Nagios and Icinga have settings where they only open events/alerts if a check failed three times in a row, because spurious failed checks are quite common. If you spam your operators with lots of alerts for monitoring checks that fix themselves, you stress the unnecessarily and create alert blindness, b…

I work on the SSO stack in a b2b company with about 200k monthly active users. One blind spot in our monitoring is when an error occurs on the client's identity provider because of a problem on our side. The service is unusable and we don't have any error logs to raise an alert. We tried to setup an alert based on expected vs actual traffic but we concluded that it would create more problems for the reason you provid…

[flagged]

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#199

Earlier quoted context omitted.

At Cloudflare’s scale on 1.1.1.1, I’d imagine you could do something comparatively simple like track ten-minute and ten-second rolling averages (I know, I know, I make that sound much easier and more practical than it actually would be), and if they differ by more than 50%, sound the alarm. (Maybe the exact numbers would need to be tweaked, e.g. 20 seconds or 80%, but it’s the idea.) Were it much less than 1.1.1.1 it…

I'm sure some engineer at cloudflare is evaluating something like this right now, and try it on historical data how many false positives that would've generated in the past, if any. Thing is, it's probably still some engineering effort, and most orgs only really improve their monitoring after it turned out to be sub-optimal.

This is hardly the first 1.1.1.1 outage. It’s also probably about the first external monitoring behaviour I imagine you’d come up with. That’s why I’m surprised—more surprised the longer I think about it, actually; more than five minutes is a really long delay to notice such a fundamental breakage.
Post reply on HN