Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

131–140 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#131

Post by bartvk »

sathackr wrote: post #130
jallmann wrote: post #9

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed…

Good writeup except the entirely false timeline shared at the beginning of the post

You need to clarify such a statement, in my opinion.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#132

Post by ahoka »

v5v3 wrote: post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

Or run your own, if you are able to.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#133

Post by sschueller »

v5v3 wrote: post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

Yes, I would also highly recommend using a DNS closest to you (for those that have ISPs that don't mess around (blocking etc.) with their DNS you usually get much better response times) and multiple from different providers.

If your device doesn't support proper failover use a local DNS forwarder on your router or an external one.

In Switzerland I would use Init7 (isp that doesn't filter) -> quad9 (unfiltered Version) -> eu dns0 (unfiltered Version)

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#134

Post by Macha »

v5v3 wrote: post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

Cloudflare's own suggested config is to use their backup server 1.0.0.1 as the secondary DNS, which was also affected by this incident.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#135

Post by sschueller »

ben0x539 wrote: post #49
0xbadcafebee wrote: post #42

Earlier quoted context omitted.

In general, the idea of DNS's design is to use the DNS resolver closest to you, rather than the one run by the largest company. That said, it's a good idea to specifically pick multiple resolvers in different regions, on different backbones, using different providers, and not use an Anycast address, because Anycast can get a little weird. However, this can lead to hard-to-troubleshoot issues, because DNS doesn't alwa…

Isn't the largest company most likely to have the DNS resolver closest to me?

No, your ISP can have a server closer before any external one.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#136

Post by rvnx »

Gieron wrote: post #88
v5v3 wrote: post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

I think normally you pair 1.1.1.1 with 1.0.0.1 and, if I understand this correctly, both were down.

8.8.8.8 + 1.1.1.1 is stable and mostly safe

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#137

Post by rvnx »

moontear wrote: post #126
Gieron wrote: post #88

Earlier quoted context omitted.

I think normally you pair 1.1.1.1 with 1.0.0.1 and, if I understand this correctly, both were down.

Just pair 1.1.1.1 with 9.9.9.9 (Quad9) so you have fault tolerance in terms of provider as well.

Quad9 is reselling the traffic logs, so it means if you connect to secret hosts (like for your work), they will be leaked

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#138

Post by chrismorgan »

philipwhiuk wrote: post #119
chrismorgan wrote: post #23

I’m surprised at the delay in impact detection: it took their internal health service more than five minutes to notice (or at least alert) that their main protocol’s traffic had abruptly dropped to around 10% of expected and was staying there. Without ever having been involved in monitoring at that kind of scale, I’d have pictured alarms firing for something that extreme within a minute. I’m curious for description o…

Remember they have no SLA for this service.

So?

They have a rather significant vested interest in it being reliable.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#139

Post by wreckage645 »

This is a good post mortem, but improvements only come with change on processes. It seems every team at CloudFlare is approaching this in isolation, without a central problem management. Every week we see a new CloudFlare global outage. It seems like the change management processes is broken and needs to be looked at..
Post reply on HN