Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

231–240 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#231
post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

3 at every place I've ever worked.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#232

Earlier quoted context omitted.

Windows 11 does not allow using this combination

You can use it, you just need to set the DNS over HTTPS templates correctly, since there's an issue with the defaults it tries to use when mixing providers. The templates you need are: 1.1.1.1: https://cloudflare-dns.com/dns-query 9.9.9.9: https://dns.quad9.net/dns-query 8.8.8.8: https://dns.google/dns-query See https://learn.microsoft.com/en-us/windows-server/networking/... for info on how to set the templates.

Awesome! Thank you!

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#233
post #224

Earlier quoted context omitted.

They are not all in the US.

Well that's the experience I had. Obviously caching was enabled (unbound), but most DNS keepalive times are so short as to be fairly useless for a single user. Even if a root server wasn't in the US, it will still be pretty slow for me. Europe is far worse. Most of Asia has bad paths to me, except for Japan and Singapore which are marginally better than the US. Maybe Aus has one...?

According to [0], there is at least one in Auckland. No idea about the veracity of that site, though.

[0] https://dnswatch.com/dns-docs/root-server-locations

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#234

This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1

dnsdist is AMAZINGLY easy to set up as a secure local resolver that forwards all queries to DoH (and checks SSL) and checks liveliness every second I need to do a write-up one day

Please do. I'd be curious what a secure-by-default self hosted resolver would look like.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#235

This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1

I don't consider these interchangeable. They have different priorities and policies. If anything I'd choose one and use my ISP default as fallback.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#236
post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

Yes, I would also highly recommend using a DNS closest to you (for those that have ISPs that don't mess around (blocking etc.) with their DNS you usually get much better response times) and multiple from different providers. If your device doesn't support proper failover use a local DNS forwarder on your router or an external one. In Switzerland I would use Init7 (isp that doesn't filter) -> quad9 (unfiltered Version…

How busy in life are you that we're concerning ourselves with nearest DNS? Are you browsing the internet like a high frequency stock trader? Seriously, in everyone's day to day, other than when these incidents happen, does someone notice a delay from resolving a domain name?

I get that in theory blah blah, but we now have choices in who gets to see all of our requests and the ISP will always lose out to the other losers in the list

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#237
post #226

Earlier quoted context omitted.

Disagree that it's fairly close to the statement "they resell traffic logs" and the implication that they leak all queried hostnames ("secret hosts, like for your work, will be leaked"). Unless Quad9 is deceiving users, both statements are, in fact, completely false. https://quad9.net/privacy/policy/#22-data-collected

>and the implication that they leak all queried hostnames ("secret hosts, like for your work, will be leaked"). The part about sharing data with "a very few carefully vetted security researchers" doesn't preclude them from leaking domains. For instance if the security researcher exports a "SELECT COUNT(*) GROUP BY hostname" query that would arguably count as "summary form", and would include any secret hostnames. > h…

I'm implying that I'm convinced they are not storing statistics on (thus leaking) every queried hostname. By your very own admission, they clearly state that they perform statistics on a set of malicious domains provided by a third party, as part of their blocking program. Additionally they publish a "top 500 domains" list regularly. You're really having a go with the shoehorn if you want "secret domains, like for your work" (read: every distinct domain queried) to fit here.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#238

Interesting to see that they probably lost 20% of 1.1.1.1 usage from a roughly 20 minute incident. Not sure how cloudflare keeps struggling with issues like these, this isn't the first (and probably won't be the last) time they have these 'simple', 'deprecated', 'legacy' issues occuring. 8.8.8.8+8.8.4.4 hasn't had a global(1) second of downtime for almost a decade. 1: localized issues did exist, but that's really the…

Then you’d be using a google DNS though which is undesirable for many.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#239
post #189
post #166

Earlier quoted context omitted.

Actually, it's about 20cm from my left elbow, which is physically several orders of magnitude closer than anything run by my ISP, and logically at least 2 network hops closer. And the closest resolving proxy DNS server for most of my machines is listening on their loopback interface. The closest such machine happens to be about 1m away, so is beaten out of first place by centimetres. (-: It's a shame that Microsoft a…

Is that really a win in terms of latency, considering that the chance of a cache hit increases with the number of users?

Keep in mind that low latency is a different goal than reliability. If you want the lowest-latency, the anycast address of a big company will often win out, because they've spent a couple million to get those numbers. If you want most reliable, then the closest hop to you should be the most reliable (there's no accounting for poor sysadmin'ing), which is often the ISP, but sometimes not.

If you run your own recursive DNS server (I keep forgetting to use the right term) on a local network, you can hit the root servers directly, which makes that the most reliable possible DNS resolver. Yes you might get more cache misses initially but I highly doubt you'd notice. (note: querying the root nameservers is bad netiquette; you should always cache queries to them for at least 5 minutes, and always use DNS resolvers to cache locally)

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#240

Earlier quoted context omitted.

Please don’t. It doesn’t make sense, doesn’t help, doesn’t improve anything and is just waste of money, time, power and people. Now without crying: I saw multiple, big companies getting rid of NOC and replacing that with on duties in multiple, focused teams. Instead of 12 people sitting 24/7 in group of 4 and doing some basic analysis and steps before calling others - you page correct people in 3-5 minutes, with exac…

24/7 on-call is basically mandatory at any major network, which cloudflare is. Your contractual relations with other networks will require it.

I'm not convinced that the SWE crowd of HN, particularly the crowd showing up to every thread about AI 'agents' really knows what it takes to run a global network or what a NOC is. I know saying this on here runs the risk of Vint Cerf or someone like that showing up in my replies, but this is seriously getting out of hand now. Every HN thread that isn't about fawning over AI companies is devolving into armchair redditor analysis of topics people know nothing about. This has gotten way worse since the pre-ChatGPT days.
Post reply on HN