Cloudflare 1.1.1.1 Incident on July 14, 2025 • Hacker News Board

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#311

Post by sophacles » Wed, Jul 16, 2025, 6:33 PM UTC

cryptonym wrote: post #55

pparanoidd wrote: post #19

Earlier quoted context omitted.

A single incident means 1.1.1.1 is no longer reasonably stable? You are the unreasonable one

I have been online for 30y and can't remember being affected by downtime from my ISP DNS. When DNS resolver is down, it affects everything, 100% uptime is a fair expectation, hence redundancy. Looks like both 1.0.0.1 and 1.1.1.1 were down for more than 1h, pretty bad TBH, especially when you advise global usage. RCA is not detailed and feels like a marketing stunt we are now getting every other week.

I too have been online for 30 years, and frequently had ISP caused dns issues, even when I wasn't using their resolvers... because of the dns interception fuckery they like to engage in. Before I started running my own resolver I saw downtime from my ISP's DNS resolver. This is across a few ISPs in that time. Anecdata is great isn't it?

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#312

Post by dylan604 » Wed, Jul 16, 2025, 6:41 PM UTC

jeffbee wrote: post #264

dylan604 wrote: post #259

Earlier quoted context omitted.

but that's the opposite of the request to move from a googDNS to a local one because of latency. so your ISP's DNS sucked, which is a broad statement, and is part of the why services like 1.1.1.1 or 8.8.8.8 exist. you didn't make the change of DNS because you were picking one based on nearest location.

There is more to latency than distance. Server response time is also important. In my case, the problem was that the DNS forwarder in the local wifi access point/router was very slow, even though the ICMP latency from my laptop to that device is obviously low.

which is well and fine, but my original comment was that moving to a closer DNS isn't worth it just for being closer especially when it is usually your ISP's server. so now, you're confirming that just moving closer isn't the solve, so it just reassures that not using the closest DNS is just fine.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#313

Post by sumtechguy » Wed, Jul 16, 2025, 6:51 PM UTC

eli wrote: post #305

karmakaze wrote: post #235

Earlier quoted context omitted.

I don't consider these interchangeable. They have different priorities and policies. If anything I'd choose one and use my ISP default as fallback.

My ISP has already been caught selling personally identifiable customer data. I trust them less than any of those companies.

My ISP one got kicked to the curb once they started returning results for anything including invalid sites. Basically to try to steer you towards their search.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#314

Post by nullify88 » Wed, Jul 16, 2025, 6:53 PM UTC

encom wrote: post #298

fragmede wrote: post #57

Earlier quoted context omitted.

Your ISP should have a DNS revolver closer to you. "Should" doesn't necessarily mean faster, however.

In case of Denmark, ISP DNS also means censored. Of course it started with CP, as it always does, then expanded to copyrights, pharmaceuticals, gambling and "terrorism". Except for the occasional Linux ISO, I don't partake in any of these topics, but I'm opposed to any kind of censorship on principle. And naturally, this doesn't stop anyone, but politicians get to stand in front of television cameras and say they're…

Not just that. ISPs are often subject to certain data retention laws. For Denmark (And other EU countries) that maybe 6 months to 2 years. And considering close ties with "9 eyes" means America potentially has access to my information anyway.

Judging by Cloudflare's privacy policy, they hold less personally identifiable information than my ISP while offering EDNS and low latencies? Win, win, win.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#315

Post by sumtechguy » Wed, Jul 16, 2025, 7:05 PM UTC

tredre3 wrote: post #284

dylan604 wrote: post #236

Earlier quoted context omitted.

How busy in life are you that we're concerning ourselves with nearest DNS? Are you browsing the internet like a high frequency stock trader? Seriously, in everyone's day to day, other than when these incidents happen, does someone notice a delay from resolving a domain name? I get that in theory blah blah, but we now have choices in who gets to see all of our requests and the ISP will always lose out to the other los…

news.ycombinator.com has a TTL of 1, so every page load will do one DNS request (possibly multiple). If you choose a resolver that is very far, 100ms longer page loads do end add up quickly...

Even something simple like www.google.com serves from 5 different DNS names. I have seen as high as 50. It is surprisingly snappier. Especially on older browsers that would only have 2 connections at a time open. It adds up faster than you would intuitively think. I used to have local resolvers that would mess with the TTL. But that was more trouble than it was worth. But it also gave a decent speedup. Was it 'worth' doing. Well it was kinda fun to mess with, I guess.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#316

Post by kingnothing » Wed, Jul 16, 2025, 7:08 PM UTC

Polizeiposaune wrote: post #277

v5v3 wrote: post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

Cloudflare recommends you configure 1.1.1.1 and 1.0.0.1 as DNS servers. Unfortunately, the configuration mistake that caused this outage disabled Cloudflare's BGP advertisements of both 1.1.1.0/24 and 1.0.0.0/24 prefixes to its peers.

A better recommendation is to use Cloudflare for one of your DNS servers and a completely different company for the other.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#317

Post by forbiddenlake » Wed, Jul 16, 2025, 7:10 PM UTC

johnklos wrote: post #220

thunderbong wrote: post #15

How does Cloudflare compare with OpenDNS?

Cloudflare is a for-profit company in the US. Their privacy claims can't be believed. Even if we did believe them, we have no idea if rsolution data isn't taken by US TLA agencies.

Hm, what distinction are you trying to make here? OpenDNS is also an American company, acquired by Cisco (an American company) in 2015.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#318

Post by toast0 » Wed, Jul 16, 2025, 7:23 PM UTC

stingraycharles wrote: post #143

Macha wrote: post #134

Earlier quoted context omitted.

Cloudflare's own suggested config is to use their backup server 1.0.0.1 as the secondary DNS, which was also affected by this incident.

TBH at this point the failure modes in which 1.1.1.1 would go down and 1.0.0.1 would not are not that many. At CloudFlare’s scale, it’s hardly believable a single of these DNS servers would go down, and it’s rather a large-scale system failure. But I understand why Cloudflare can’t just say “use 8.8.8.8 as your backup”.

It would depend on how Cloudflare set up their systems. From this and other outages, I think it's pretty clear that they've set up their systems as a single failure domain. But it would be possible for them to have setup for 1.1.1.1 and 1.0.0.1 to have separate failure domains --- separate infrastructure, at least some sites running one but not the other.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#319

Post by outworlder » Wed, Jul 16, 2025, 7:27 PM UTC

karmakaze wrote: post #235

homebrewer wrote: post #180

This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1

I don't consider these interchangeable. They have different priorities and policies. If anything I'd choose one and use my ISP default as fallback.

My ISP (one of the largest in the US) like to hijack DNS responses (specially NXDOMAIN) and serve crap. No thanks. Which is also why I have to use encryption to talk to public DNS servers otherwise they will hijack anyways.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#320

Post by zamadatix » Wed, Jul 16, 2025, 7:35 PM UTC

kachapopopow wrote: post #112

Interesting to see that they probably lost 20% of 1.1.1.1 usage from a roughly 20 minute incident. Not sure how cloudflare keeps struggling with issues like these, this isn't the first (and probably won't be the last) time they have these 'simple', 'deprecated', 'legacy' issues occuring. 8.8.8.8+8.8.4.4 hasn't had a global(1) second of downtime for almost a decade. 1: localized issues did exist, but that's really the…

Regarding the 20% some clients/resolvers will mark a server as temporarily down if it fails to respond to multiple queries in a row. That way the user doesn't have to wait the timeout delay 500 times in a row on the next 500 queries.

From the longer term graphs it looks like volume returned to normal https://imgur.com/a/8a1H8eL