Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

351–360 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#351

Earlier quoted context omitted.

There are no good private DNS configurations, but if you don't trust the big caching recursive resolvers then I'd consider just running your own at home. Unbound is easy to set up and you'll probably never notice a speed difference.

I trust my isp far more than I trust cloudflare and google

Why? Some were injecting ads, blocking services, degrading video and other wrongdoings.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#352

Interesting to see that they probably lost 20% of 1.1.1.1 usage from a roughly 20 minute incident. Not sure how cloudflare keeps struggling with issues like these, this isn't the first (and probably won't be the last) time they have these 'simple', 'deprecated', 'legacy' issues occuring. 8.8.8.8+8.8.4.4 hasn't had a global(1) second of downtime for almost a decade. 1: localized issues did exist, but that's really the…

You’re not sure how they’re struggling to fix an engineering problem characterized by complexity and scale encountered by 0.001% of network engineers?

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#353

I’m surprised at the delay in impact detection: it took their internal health service more than five minutes to notice (or at least alert) that their main protocol’s traffic had abruptly dropped to around 10% of expected and was staying there. Without ever having been involved in monitoring at that kind of scale, I’d have pictured alarms firing for something that extreme within a minute. I’m curious for description o…

The service almost certainly wasn't completely hard down at the time the impact began, especially if that's the start of a global rollout. It would have taken time for the impact to become measurable.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#354

Interesting that traffic didn't return to completely normal levels after the incident. I recently started using the "luci-app-https-dns-proxy" package on OpenWrt, which is preconfigured to use both Cloudflare and Google DNS, and since DoH was mostly unaffected, I didn't notice an outage. (Though if DoH had been affected, it presumably would have failed over to Google DNS anyway.)

If your Internet doesn't work, you'll get up and do other things for a while. I strongly suspect most folks didn't switch DNS providers in that time.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#355

This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1

I think systemd-resolved does something similar if you use that. Does DoT and DNSSEC by default.

If you want to eschew centralized DNS altogether, if you run a Tor daemon, it has an option to expose a DNS resolver to your network. Multiple resolvers if you want them.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#356

Earlier quoted context omitted.

Realistically, either you ignore the privacy concerns and set up routing to multiple providers preferring the fastest, or you go all-in on privacy and route DNS over Tor over bridge. Although, perhaps, having an external VPS with a dns proxy could be a good middle ground?

Middle ground is ISP DNS, right?

If privacy is your primary concern I would 100% trust Cloudflare or Google over an ISP in the US

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#357
post #324

Earlier quoted context omitted.

Ah, this is for DoH, gotcha! This "URL template" thing seems odd – is Windows doing something like creating a URL out of the DNS IP and a pattern, e.g. 1.1.1.1 + " /foo" rel="nofollow">https:// /foo " would yield https://1.1.1.1/foo ? If so, why not just allow providing an actual URL for each server?

It does allow you to provide a URL for each server. The issue is just that its default behavior doesn't work for all providers. I have another comment in this thread telling the original commenter how to configure it.

Very cool, thank you!

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#358

Earlier quoted context omitted.

Cloudflare recommends you configure 1.1.1.1 and 1.0.0.1 as DNS servers. Unfortunately, the configuration mistake that caused this outage disabled Cloudflare's BGP advertisements of both 1.1.1.0/24 and 1.0.0.0/24 prefixes to its peers.

A better recommendation is to use Cloudflare for one of your DNS servers and a completely different company for the other.

Just wondering, how do y'all manage wifi portals and manually setting DNS services? I used to use cf and google's but it was so annoying to disable and re-enable that every time I use a public wifi network.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#359
post #351

Earlier quoted context omitted.

I trust my isp far more than I trust cloudflare and google

Why? Some were injecting ads, blocking services, degrading video and other wrongdoings.

Maybe their ISPs don't do that. There are many ISPs on the Earth.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#360
post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

If you think you can pontificate on DNS then I think you should be running your own service.

Note how root "." just works and has done for decades - that's proper engineering and actually way more complicated than running 1.1.1.1. What 1.1.1.1 suffers from is anycast and not DNS.

Cloudflare (and Google and co) insist on using one or more "vanity" IP addresses - that is very unfair of me but that it what it is, and to make it work, they have to use anycast.

The real issue is fixing anycast and not DNS.

Anyway, select two+ providers and set them.

Post reply on HN