Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

61–70 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#61

Post by codingminds »

komali2 wrote: post #29
codingminds wrote: post #18

Earlier quoted context omitted.

If you consume a service that's free of charge, it's at least not reasonable to complain if there's an outage. Like mentioned by other comments, do it on your own if you are not happy with the stability. Or just pay someone to provide it - like your ISP.. And TBH I trust my local ISP more than Google or CF. Not in availability, but it's covered by my local legislature. That's a huge difference - in a positive way.

> it's at least not reasonable to complain if there's an outage. I don't think this is fair when discussing infrastructure. It's reasonable to complain about potholes, undrinkable tap water, long lines at the DMV, cracked (or nonexistent) sidewalks, etc. The internet is infrastructure and DNS resolution is a critical part of it. That it hasn't been nationalized doesn't change the fact that it's infrastructure (and ac…

You are right infrastructure is important.

But opposite to tap water there are a lot of different free DNS resolvers that can be used.

And I don't see how my taxes funded CFs DNS service. But my ISP fee covers their DNS resolving setup. That's the reason why I wrote

> a service that's free of charge

Which CF is.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#62

Post by misiek08 »

mentalgear wrote: post #40
TheDong wrote: post #39

Earlier quoted context omitted.

I'm not surprised. Let's say you've got a metric aggregation service, and that service crashes. What does that result in? Metrics get delayed until your orchestration system redeploys that service elsewhere, which looks like a 100% drop in metrics. Most orchestration take a sec to redeploy in this case, assuming that it could be a temporary outage of the node (like a network blip of some sort). Sooo, if you alert aft…

Its not wrong for smaller companies. But there's an argument that a big system critical company/provider like Cloudflare should be able to afford its own always on team with a night shift.

Please don’t. It doesn’t make sense, doesn’t help, doesn’t improve anything and is just waste of money, time, power and people.

Now without crying: I saw multiple, big companies getting rid of NOC and replacing that with on duties in multiple, focused teams. Instead of 12 people sitting 24/7 in group of 4 and doing some basic analysis and steps before calling others - you page correct people in 3-5 minutes, with exact and specific alert.

Incident resolution times went greatly down (2-10x times - depends on company), people don’t have to sit overnight and sleep for most of the time and no stupid actions like service restart taken to slow down incident resolution.

And I’m not liking that some platforms hire 1500 people for job that could be done with 50-100, but in terms of incident response - if you already have teams with separated responsibilities then NOC it’s "legacy"

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#64

Post by misiek08 »

nojs wrote: post #41
sammy2255 wrote: post #8

Earlier quoted context omitted.

Depends who you trust more with your DNS traffic. I know who I trust more.

Who? Honest question

Google is serving you ads, CF isn’t.

And it’s not conspiracy theory - it was very suspicious when we did some testing on small, aware group. The traffic didn’t look like being handled anonymously at Google side

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#65

Post by Elucalidavah »

nojs wrote: post #41
sammy2255 wrote: post #8

Earlier quoted context omitted.

Depends who you trust more with your DNS traffic. I know who I trust more.

Who? Honest question

Realistically, either you ignore the privacy concerns and set up routing to multiple providers preferring the fastest, or you go all-in on privacy and route DNS over Tor over bridge.

Although, perhaps, having an external VPS with a dns proxy could be a good middle ground?

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#66

Post by mnordhoff »

misiek08 wrote: post #64
nojs wrote: post #41

Earlier quoted context omitted.

Who? Honest question

Google is serving you ads, CF isn’t. And it’s not conspiracy theory - it was very suspicious when we did some testing on small, aware group. The traffic didn’t look like being handled anonymously at Google side

Unless the privacy policy changed recently, Google shouldn't be doing anything nefarious with 8.8.8.8 DNS queries.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#67

Post by willejs »

sneak wrote: post #43
jallmann wrote: post #9

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed…

I disagree. The actual root cause here is shrouded in jargon that even experienced admins such as myself have to struggle to parse. It’s corporate newspeak. “legacy” isn’t a clear term, it’s used to abstract and obfuscate. > Legacy components do not leverage a gradual, staged deployment methodology. Cloudflare will deprecate these systems which enables modern progressive and health mediated deployment processes to pr…

If you carry on reading, its quite obvious they misconfigured a service and routed production traffic to that instead of the correct service, and the system used to do that was built in 2018 and is considered legacy (probably because you can easily deploy bad configs). Given that, I wouldn't say the summary is "inscrutable corporatese" whatever that is.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#68

Post by stingraycharles »

bauruine wrote: post #17
jallmann wrote: post #9

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed…

How does DoH work? Somehow you need to know the IP of cloudflare-dns.com first. Maybe your router uses 1.1.1.1 for this.

Yeah I don’t understand this part either, maybe it’s supposed to be bootstrapped using your ISP’s DNS server?

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#69

Post by LeoPanthera »

dontTREATonme wrote: post #54
0xbadcafebee wrote: post #42

Earlier quoted context omitted.

In general, the idea of DNS's design is to use the DNS resolver closest to you, rather than the one run by the largest company. That said, it's a good idea to specifically pick multiple resolvers in different regions, on different backbones, using different providers, and not use an Anycast address, because Anycast can get a little weird. However, this can lead to hard-to-troubleshoot issues, because DNS doesn't alwa…

What’s your recommendation for finding the dns resolver closest to me? I currently use 1.1 and 8.8, but I’m absolutely open to alternatives.

The closest DNS resolver to you is the one run by your ISP.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#70

Post by Pharaoh2 »

angst wrote: post #11

I wonder how uptime ratio of 1.1.1.1 is against 8.8.8.8 Maybe there is noticeable difference? I have seen more outage incident reports of cloudflare than of google, but this is just personal anecdote.

https://www.dnsperf.com/#!dns-resolvers

Last 30 days, 8.8.8.8 has 99.99% uptime vs 1.1.1.1 has 99.09%

Post reply on HN