Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

111–120 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#111

Post by komali2 »

codingminds wrote: post #61
komali2 wrote: post #29

Earlier quoted context omitted.

> it's at least not reasonable to complain if there's an outage. I don't think this is fair when discussing infrastructure. It's reasonable to complain about potholes, undrinkable tap water, long lines at the DMV, cracked (or nonexistent) sidewalks, etc. The internet is infrastructure and DNS resolution is a critical part of it. That it hasn't been nationalized doesn't change the fact that it's infrastructure (and ac…

You are right infrastructure is important. But opposite to tap water there are a lot of different free DNS resolvers that can be used. And I don't see how my taxes funded CFs DNS service. But my ISP fee covers their DNS resolving setup. That's the reason why I wrote > a service that's free of charge Which CF is.

DNS shouldn't be privatized at all since it's a critical part of internet infrastructure, however at the same time the idea that somehow it's something a corporation should be allowed to sell to you at all (or "give you for free") is silly given that the service is meaningless without the infrastructure of the internet, which is built by governments (through taxes). I can't even think of an equivalent it's so ridiculous that it's allowed at all, my best guess would be maybe, if your landlord was allowed to charge you for walking on the sidewalk in front of the apartment or something.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#112

Post by kachapopopow »

Interesting to see that they probably lost 20% of 1.1.1.1 usage from a roughly 20 minute incident.

Not sure how cloudflare keeps struggling with issues like these, this isn't the first (and probably won't be the last) time they have these 'simple', 'deprecated', 'legacy' issues occuring.

8.8.8.8+8.8.4.4 hasn't had a global(1) second of downtime for almost a decade.

1: localized issues did exist, but that's really the fault of the internet and they did remain running when google itself suffered severe downtime in various different services.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#113

Post by Algent »

Gieron wrote: post #88
v5v3 wrote: post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

I think normally you pair 1.1.1.1 with 1.0.0.1 and, if I understand this correctly, both were down.

Yeah pretty much. In a perfect world you would pair it with another service I guess but usually you use the official backup IP because it's not supposed to break at same time.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#114

Post by bigiain »

chrismorgan wrote: post #109
bigiain wrote: post #98

Earlier quoted context omitted.

There are kinds big step/jumps as the size of a company goes up. Step 1: You start out with the founders being on call 27x7x365 or people in the first 10 or 20 hires "carry the pager" on weekends and evenings and your entire company is doing unpaid rostered on call. Step 2: You steal all the underwear. Step 3: You have follow-the-sun office-hours support staff teams distributed around the globe with sufficient covera…

I confess myself bemused by your Step 2.

I'm like, come on! It's a South Park reference? Surely everybody here gets that???

"Original air date: December 16, 1998"

Oh, right. Half of you weren't even born... Now I feel ooooooold.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#115

Post by Algent »

radicaldreamer wrote: post #7
caconym_ wrote: post #4

Earlier quoted context omitted.

> Interesting that traffic didn't return to completely normal levels after the incident. Anecdotally, I figured out their DNS was broken before it hit their status page and switched my upstream DNS over to Google. Haven't gotten around to switching back yet.

What would be a good reason to switch back from Google DNS?

After trying both several time I since stayed with google due to cloudflare always returning really bad IPs for anything involving CDN. Having users complain stuff take age to load because you got matched to an IP on opposite side of planet is a bit problematic especially when it rarely happen on other dns providers. Maybe there is a way to fix this but I admit I went for the easier option of going back to good old 8.8.8.8

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#116

Post by philipwhiuk »

perlgeek wrote: post #93

An outage of roughly 1 hour is 0.13% of a month or 0.0114% of a year. It would be interesting to see the service level objective (SLO) that cloudflare internally has for this service. I've found https://www.cloudflare.com/r2-service-level-agreement/ but this seems to be for payed services, so this outage would put July in the " = 99.0%" bucket, so you'd get a 10% refund for the month if you payed for it.

Probably 99.9% or better annually just from a 'maintaining reputation for reliability' standpoint.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#117

Post by nness »

Interesting side-effect, the Gluetun docker image uses 1.1.1.1 for DNS resolution — as a result of the outage Gluetun's health checks failed and the images stopped.

If there were some way to view torrenting traffic, no doubt there'd be a 20 minute slump.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#118

Post by [deleted] »

ben0x539 wrote: post #49
0xbadcafebee wrote: post #42

Earlier quoted context omitted.

In general, the idea of DNS's design is to use the DNS resolver closest to you, rather than the one run by the largest company. That said, it's a good idea to specifically pick multiple resolvers in different regions, on different backbones, using different providers, and not use an Anycast address, because Anycast can get a little weird. However, this can lead to hard-to-troubleshoot issues, because DNS doesn't alwa…

Isn't the largest company most likely to have the DNS resolver closest to me?

[deleted]

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#119

Post by philipwhiuk »

chrismorgan wrote: post #23

I’m surprised at the delay in impact detection: it took their internal health service more than five minutes to notice (or at least alert) that their main protocol’s traffic had abruptly dropped to around 10% of expected and was staying there. Without ever having been involved in monitoring at that kind of scale, I’d have pictured alarms firing for something that extreme within a minute. I’m curious for description o…

Remember they have no SLA for this service.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#120

Post by lillecarl »

Algent wrote: post #113
Gieron wrote: post #88

Earlier quoted context omitted.

I think normally you pair 1.1.1.1 with 1.0.0.1 and, if I understand this correctly, both were down.

Yeah pretty much. In a perfect world you would pair it with another service I guess but usually you use the official backup IP because it's not supposed to break at same time.

I would rather fall back to the slow path of resolving through root servers than fall back from one recursive resolver to another.
Post reply on HN