Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

21–30 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#21

Post by nodesocket »

I used to configure 1.1.1.1 as primary and 8.8.8.8 as secondary but noticed that Cloudflare on aggregate was quicker to respond to queries and changed everything to use 1.1.1.1 and 1.0.0.1. Perhaps I'll switch back to using 8.8.8.8 as secondary, though my understanding is DNS will round-robin between primary and secondary, it's not primary and then use secondary ONLY if primary is down. Perhaps I am wrong though.

EDIT: Appears I was wrong, it is failover not round-robin between the primary and secondary DNS servers. Thus, using 1.1.1.1 and 8.8.8.8 makes sense.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#22

Post by yjftsjthsd-h »

sammy2255 wrote: post #10
CuteDepravity wrote: post #5

It's crazy that both 1.1.1.1 and 1.0.0.1 where affected by the same change I guess now we should start using a completely different provider as dns backup Maybe 8.8.8.8 or 9.9.9.9

1.1.1.1 and 1.0.0.1 are served by the same service. It's not advertised as a redundant fully separate backup or anything like that...

Wait, then why does 1.0.0.1 exist? I'll grant I've never seen it advertised/documented as a backup, but I just assumed it must be because why else would you have two? (Given that 1.1.1.1 already isn't actually a single point, so I wouldn't think you need a second IP for load balancing reasons.)

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#23

Post by chrismorgan »

I’m surprised at the delay in impact detection: it took their internal health service more than five minutes to notice (or at least alert) that their main protocol’s traffic had abruptly dropped to around 10% of expected and was staying there. Without ever having been involved in monitoring at that kind of scale, I’d have pictured alarms firing for something that extreme within a minute. I’m curious for description of how and why that might be, and whether it’s reasonable or surprising to professionals in that space too.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#24

Post by yjftsjthsd-h »

pparanoidd wrote: post #19
geoffpado wrote: post #6

This was quite annoying for me, having only switched my DNS server to 1.1.1.1 approximately 3 weeks ago to get around my ISP having a DNS outage. Is reasonably stable DNS really so much to ask for these days?

A single incident means 1.1.1.1 is no longer reasonably stable? You are the unreasonable one

Although I agree 1.1.1.1 is fine: To this particular commenter they've had one major outage in 3 total weeks of use, which isn't exactly a good record. (And it's understandable to weigh personal experience above other people claiming this isn't representative.)

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#25

Post by ta1243 »

bauruine wrote: post #17
jallmann wrote: post #9

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed…

How does DoH work? Somehow you need to know the IP of cloudflare-dns.com first. Maybe your router uses 1.1.1.1 for this.

And even if you have already resolved it the TTL is only 5 minutes

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#26

Post by [deleted] »

bauruine wrote: post #17
jallmann wrote: post #9

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed…

How does DoH work? Somehow you need to know the IP of cloudflare-dns.com first. Maybe your router uses 1.1.1.1 for this.

[deleted]

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#27

Post by ta1243 »

yjftsjthsd-h wrote: post #22
sammy2255 wrote: post #10

Earlier quoted context omitted.

1.1.1.1 and 1.0.0.1 are served by the same service. It's not advertised as a redundant fully separate backup or anything like that...

Wait, then why does 1.0.0.1 exist? I'll grant I've never seen it advertised/documented as a backup, but I just assumed it must be because why else would you have two? (Given that 1.1.1.1 already isn't actually a single point, so I wouldn't think you need a second IP for load balancing reasons.)

Far quicker to type ping 1.1 than ping 1.1.1.1

1.0.0.0/24 is a different network than 1.1.1.0/24 too, so can be hosted elsewhere. Indeed right now 1.1.1.1 from my laptop goes via 141.101.71.63 and 1.0.0.1 via 141.101.71.121, which are both hosts on the same LINX/LON1 peer but presumably from different routers, so there is some resilience there.

Given DNS is about the easiest thing to avoid a single point of failure on I'm not sure why you would put all your eggs in a single company, but that seems to be the modern internet - centralisation over resilience because resilience is somehow deemed to be hard.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#28

Post by nelox »

bauruine wrote: post #17
jallmann wrote: post #9

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed…

How does DoH work? Somehow you need to know the IP of cloudflare-dns.com first. Maybe your router uses 1.1.1.1 for this.

[flagged]

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#29

Post by komali2 »

codingminds wrote: post #18
geoffpado wrote: post #6

This was quite annoying for me, having only switched my DNS server to 1.1.1.1 approximately 3 weeks ago to get around my ISP having a DNS outage. Is reasonably stable DNS really so much to ask for these days?

If you consume a service that's free of charge, it's at least not reasonable to complain if there's an outage. Like mentioned by other comments, do it on your own if you are not happy with the stability. Or just pay someone to provide it - like your ISP.. And TBH I trust my local ISP more than Google or CF. Not in availability, but it's covered by my local legislature. That's a huge difference - in a positive way.

> it's at least not reasonable to complain if there's an outage.

I don't think this is fair when discussing infrastructure. It's reasonable to complain about potholes, undrinkable tap water, long lines at the DMV, cracked (or nonexistent) sidewalks, etc. The internet is infrastructure and DNS resolution is a critical part of it. That it hasn't been nationalized doesn't change the fact that it's infrastructure (and access absolutely should be free) and therefore everyone should feel free to complain about it not working correctly.

"But you pay taxes for drinkable tap water," yes, and we paid taxes to make the internet work too. For some reason, some governments like the USA feel it to be a good idea to add a middle man to spend that tax money on, but, fine, we'll complain about the middle man then as well.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#30

Post by yjftsjthsd-h »

ta1243 wrote: post #27
yjftsjthsd-h wrote: post #22

Earlier quoted context omitted.

Wait, then why does 1.0.0.1 exist? I'll grant I've never seen it advertised/documented as a backup, but I just assumed it must be because why else would you have two? (Given that 1.1.1.1 already isn't actually a single point, so I wouldn't think you need a second IP for load balancing reasons.)

Far quicker to type ping 1.1 than ping 1.1.1.1 1.0.0.0/24 is a different network than 1.1.1.0/24 too, so can be hosted elsewhere. Indeed right now 1.1.1.1 from my laptop goes via 141.101.71.63 and 1.0.0.1 via 141.101.71.121, which are both hosts on the same LINX/LON1 peer but presumably from different routers, so there is some resilience there. Given DNS is about the easiest thing to avoid a single point of failure o…

> Far quicker to type ping 1.1 than ping 1.1.1.1

I guess. I wouldn't have thought it worthwhile for 4 chars, but yes.

> 1.0.0.0/24 is a different network than 1.1.1.0/24 too, so can be hosted elsewhere.

I thought anycast gave them that on a single IP, though perhaps this is even more resilient?

Post reply on HN