Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

161–170 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#161
post #137

Earlier quoted context omitted.

Just pair 1.1.1.1 with 9.9.9.9 (Quad9) so you have fault tolerance in terms of provider as well.

Quad9 is reselling the traffic logs, so it means if you connect to secret hosts (like for your work), they will be leaked

Is this true? They claim that they don't keep any logs. Do you have a source?

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#162
post #134

Earlier quoted context omitted.

Cloudflare's own suggested config is to use their backup server 1.0.0.1 as the secondary DNS, which was also affected by this incident.

TBH at this point the failure modes in which 1.1.1.1 would go down and 1.0.0.1 would not are not that many. At CloudFlare’s scale, it’s hardly believable a single of these DNS servers would go down, and it’s rather a large-scale system failure. But I understand why Cloudflare can’t just say “use 8.8.8.8 as your backup”.

At least some machines/routers do NOT have a primary and backup but instead randomly round-robin between them.

Which means that you’d be on cloudflare half the time and on google half the time which may not be what you wanted.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#163
post #85

Earlier quoted context omitted.

That's not correct. LetEncrypt are trialling ip address https/TLS certificates right now: https://letsencrypt.org/2025/07/01/issuing-our-first-ip-addr... They say: "In principle, there’s no reason that a certificate couldn’t be issued for an IP address rather than a domain name, and in fact the technical and policy standards for certificates have always allowed this, with a handful of certificate authorities offering…

right, this was announced about two weeks ago to some fanfare. So in principle there was no reason not to do it two decades ago? It would've been nice back then. I never heard of any certificate authority offering that.

It the beginning of HTTPS you were supposed to look for the padlock to prove if was a safe site. Scammers wouldn’t take the time and money to get a cert, after all!

So certs were often tied with identity which an IP really isn’t so few providers offered them.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#164

I’m surprised at the delay in impact detection: it took their internal health service more than five minutes to notice (or at least alert) that their main protocol’s traffic had abruptly dropped to around 10% of expected and was staying there. Without ever having been involved in monitoring at that kind of scale, I’d have pictured alarms firing for something that extreme within a minute. I’m curious for description o…

This is one of those graphs that would have been on the giant wall in the NOC in the old days - someone would glance up and see it had dropped and say “that’s not right” and start scrambling.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#165

Earlier quoted context omitted.

Private DNS on Android refers to 'DNS over HTTPS' and would normally only accept a hostname. Normal DNS can normally be changed in your connection settings for a given connection on most flavours of Android.

Its DNS over TLS. Android does not support DNS over HTTPS except Google's DNS

As far as I understand it, it's Google or Cloudflare?

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#166

Earlier quoted context omitted.

What’s your recommendation for finding the dns resolver closest to me? I currently use 1.1 and 8.8, but I’m absolutely open to alternatives.

The closest DNS resolver to you is the one run by your ISP.

Actually, it's about 20cm from my left elbow, which is physically several orders of magnitude closer than anything run by my ISP, and logically at least 2 network hops closer.

And the closest resolving proxy DNS server for most of my machines is listening on their loopback interface. The closest such machine happens to be about 1m away, so is beaten out of first place by centimetres. (-:

It's a shame that Microsoft arbitrarily ties such functionality to the Server flavour of Windows, and does not supply it on the Workstation flavour, but other operating systems are not so artificially limited or helpless; and even novice users on such systems can get a working proxy DNS server out of the box that their sysops don't actually have to touch.

The idea that one has to rely upon an ISP, or even upon CloudFlare and Google and Quad9, for this stuff is a bit of a marketing tale that is put about by thse self-same ISPs and CloudFlare and Google and Quad9. Not relying upon them is not actually limited to people who are skilled in system operation, i.e. who they are; but rather merely limited by what people run: black box "smart" tellies and whatnot, and the Workstation flavour of Microsoft Windows. Even for such machines, there's the option of a decent quality router/gateway or simply a small box providing proxy DNS on the LAN.

In my case, said small box is roughly the size of my hand and is smaller than my mass-market SOHO router/gateway. (-:

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#167

Earlier quoted context omitted.

There's a constant tension between speed of detection and false positive rates. Traditional monitoring systems like Nagios and Icinga have settings where they only open events/alerts if a check failed three times in a row, because spurious failed checks are quite common. If you spam your operators with lots of alerts for monitoring checks that fix themselves, you stress the unnecessarily and create alert blindness, b…

At Cloudflare’s scale on 1.1.1.1, I’d imagine you could do something comparatively simple like track ten-minute and ten-second rolling averages (I know, I know, I make that sound much easier and more practical than it actually would be), and if they differ by more than 50%, sound the alarm. (Maybe the exact numbers would need to be tweaked, e.g. 20 seconds or 80%, but it’s the idea.) Were it much less than 1.1.1.1 it…

I would want to make sure we avoid "We should always do the exact specific thing that would have prevented this exact specific issue"-style thinking.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#168
post #85

Earlier quoted context omitted.

That's not correct. LetEncrypt are trialling ip address https/TLS certificates right now: https://letsencrypt.org/2025/07/01/issuing-our-first-ip-addr... They say: "In principle, there’s no reason that a certificate couldn’t be issued for an IP address rather than a domain name, and in fact the technical and policy standards for certificates have always allowed this, with a handful of certificate authorities offering…

right, this was announced about two weeks ago to some fanfare. So in principle there was no reason not to do it two decades ago? It would've been nice back then. I never heard of any certificate authority offering that.

> I never heard of any certificate authority offering that.

DigiCert does. That is where 1.1.1.1 and 9.9.9.9 get their valid certificates from

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#169
post #137

Earlier quoted context omitted.

Just pair 1.1.1.1 with 9.9.9.9 (Quad9) so you have fault tolerance in terms of provider as well.

Quad9 is reselling the traffic logs, so it means if you connect to secret hosts (like for your work), they will be leaked

Could you show a citation? Your statement completely opposes Quad9's official information as published on quad9.net, and what's more it doesn't align at all with Bill Woodcock's known advocacy for privacy.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#170

It's crazy that both 1.1.1.1 and 1.0.0.1 where affected by the same change I guess now we should start using a completely different provider as dns backup Maybe 8.8.8.8 or 9.9.9.9

In general there's no such thing as "DNS backup". Most clients just arbitrarily pick one from the list, they don't fall back to the other one in case of failure or anything. So if one went down you'd still find many requests timing out.

The reality is that it's rather complicated to say what "most clients" do, as there is some behavioural variation amongst the DNS client libraries when they are configured with multiple IP addresses to contact. So whilst it's true to say that fallback and redundancy does not always operate as one might suppose at the DNS client level, it is untrue to go to the opposite extreme and say that there's no such thing at all.
Post reply on HN