Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

371–380 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#371

Post by wlonkly »

gerdesj wrote: post #360
v5v3 wrote: post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

If you think you can pontificate on DNS then I think you should be running your own service. Note how root "." just works and has done for decades - that's proper engineering and actually way more complicated than running 1.1.1.1. What 1.1.1.1 suffers from is anycast and not DNS. Cloudflare (and Google and co) insist on using one or more "vanity" IP addresses - that is very unfair of me but that it what it is, and to…

The root servers all use anycast addresses.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#372

Post by genewitch »

adornKey wrote: post #341
genewitch wrote: post #258

Earlier quoted context omitted.

i am 95% certain i run unbound in a datacenter, and i have pihole local, my PC connects to pihole first, and if that's down, it connects to my DC; pihole connects to the DC and one of the filtered DNS providers (don't remember which) and GTEi's old server, that still works and has never let me down. No, not that one, the other one. i have musknet, though, so i can't edit the DNS providers on the router without buying…

Having a 2nd trustworthy router consumes extra energy, but maybe it's worth it. More than once my router made an update and silently disabled the pi-hole. Having a fully configured spare pi-hole in a box also helps. Another time my pi-hole refused to boot after a power outage.

well, i completely agree. I'm the author of a couple of "how-to run ipcop / monowall in " articles on broadbandreports. So of course, when i heard i can get real, honest to goodness, publicly routable ipv6 on starlink with a third party router, i set to making one in proxmox local, here. None of the "router distributions" worked correctly, if at all, and none of them that i tried had ipv6 settings anywhere.

So i went to best buy and bought 3 routers, and set each one up for 1 week. Turns out, you can get public routable ipv6 with a third party router, if the router supports ipv6.

I still see people mentioning opnsense and pfsense on here from time to time, and i wonder if i got the wrong - maybe outdated - iso images? I also tried doing it with freebsd and debian and couldn't figure it out, which is a bit depressing for me. I'll try again someday.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#373

Post by kentonv »

zac23or wrote: post #302
kentonv wrote: post #268

Earlier quoted context omitted.

> some technical decisions are absurd, such as the worker's cache.delete method, which only clears the cache contents in the data center where the Worker was invoked!!! The Cache API is a standard taken from browsers. In the browser, cache.delete obviously only deletes that browser's cache, not all other browsers in the world. You could certainly argue that a global purge would be more useful in Workers, but it would…

>cache.delete obviously only deletes that browser's cache, not all other browsers in the world. To me, it only makes sense if the put method creates a cache only in the datacenter where the Worker was invoked. Put and delete need to be related, in my opinion. Now I'm curious: what's the point of clearing the cache contents in the datacenter where the Worker was invoked? I can't think of any use for this method. My cr…

> To me, it only makes sense if the put method creates a cache only in the datacenter where the Worker was invoked. Put and delete need to be related, in my opinion.

That is, in fact, how it works. cache.put() only writes to the local datacenter's cache. If delete() were global, it would be inconsistent with put().

> Now I'm curious: what's the point of clearing the cache contents in the datacenter where the Worker was invoked? I can't think of any use for this method.

Say you read the cache entry but you find, based on its content, that it is no longer valid. You would then want to delete it, to save the cost of reading it again later.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#374

Post by bauruine »

ripdog wrote: post #210
xpe wrote: post #201

Earlier quoted context omitted.

Ouch. What resolver? What hardware? With something like a N100- or N150-based single board computer (perhaps around $200) running any number of open source DNS resolvers, I would expect you can average around 30 ms for cold lookups and <1 ms for cache hits.

Not a hardware issue, but a physics problem. I live in NZ. I guess the root servers are all in the US, so that's 130ms per trip minimum.

The root servers aren't the problem. They are heavily anycasted and i'm sure there are many in .nz. If that was the issue you could simply serve the root zone yourself, at least some of them allow axfr. [0] This info is also easy cacheable, they have big TTLs and you only have to do it once for each tld. The authoritative name server of the domain you want to access on the other hand are often just in the US or Europe and are the main issue.

Edit: How to serve the root zone locally with unbound. https://old.reddit.com/r/pihole/comments/s43o8j/where_does_u...

[0] dig axfr . @k.root-servers.net

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#375

Post by kentonv »

freedomben wrote: post #293
kentonv wrote: post #282

Earlier quoted context omitted.

> and when they make a change that breaks the existing API contract it always makes for several miserable days for me If we changed an API in Workers in a way that broke any Worker in production, we consider that an incident and we will roll it back ASAP. We really try to avoid this but sometimes it's hard for us to tell. Please feel free to contact us if this happens in the future (e.g. file a support ticket or file…

Thank you! To clarify it's been API contracts in the DNS record setting API that have hit me. I'm going from memory here and it's been a couple years I think so might be a bit rusty, but one example was a slight change in data type acceptance for TTL on a record. It used to take either a string or integer in the JSON but at some point started rejecting integers (or strings, whichever one I was sending at the time sto…

Hmm, I am not familiar with the DNS API but those certainly sound like changes that I'd argue should not have been made. :/

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#376

Post by ripdog »

bauruine wrote: post #374
ripdog wrote: post #210

Earlier quoted context omitted.

Not a hardware issue, but a physics problem. I live in NZ. I guess the root servers are all in the US, so that's 130ms per trip minimum.

The root servers aren't the problem. They are heavily anycasted and i'm sure there are many in .nz. If that was the issue you could simply serve the root zone yourself, at least some of them allow axfr. [0] This info is also easy cacheable, they have big TTLs and you only have to do it once for each tld. The authoritative name server of the domain you want to access on the other hand are often just in the US or Europ…

Thank you for the correction, I did get that wrong. To be clear, there was no easy solution to get reliable, low latency DNS responses from my own resolver without breaking keepalive by forcibly caching entries longer?

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#377

Post by johnklos »

astrange wrote: post #364
johnklos wrote: post #349

Earlier quoted context omitted.

It seems we have a lot of Cloudflare fanbois and apologists here. This is not unexpected. But is anything I'm writing untrue, or just unpopular? Does anyone who's downvoting me care to point out any inaccuracies about what I've written?

It's illegal for a US company to lie to their investors, so if you believe they're lying to you, you should sue them for securities fraud.

You actually believe that this is how it works?

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#378

Post by bauruine »

ripdog wrote: post #376
bauruine wrote: post #374

Earlier quoted context omitted.

The root servers aren't the problem. They are heavily anycasted and i'm sure there are many in .nz. If that was the issue you could simply serve the root zone yourself, at least some of them allow axfr. [0] This info is also easy cacheable, they have big TTLs and you only have to do it once for each tld. The authoritative name server of the domain you want to access on the other hand are often just in the US or Europ…

Thank you for the correction, I did get that wrong. To be clear, there was no easy solution to get reliable, low latency DNS responses from my own resolver without breaking keepalive by forcibly caching entries longer?

Not that I know of except from having a big cache and many users that keeps it warm. As I said you could run a local root zone but that only saves you the one time lookup every week+ of the tld name servers and the root servers are generally very close to you. There is a map of all root servers. There are 12 in .nz alone. A few cc tlds are providing their zone via axfr [1] so you could add that to your resolver to save some roundtrips but I don't think having .ch or .se locally will make a big difference and they are 1.2GB each and you would need to download them daily.

[0]: https://root-servers.org/ [1]: https://github.com/jschauma/tld-zoneinfo

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#379

Post by hdgvhicv »

bagels wrote: post #351
hdgvhicv wrote: post #350

Earlier quoted context omitted.

I trust my isp far more than I trust cloudflare and google

Why? Some were injecting ads, blocking services, degrading video and other wrongdoings.

Mine doesn’t do that, mine is very transparent about what they do, what they will support, what laws they have to follow, what guidelines they can ignore, what logging they do, and if I have issues I jump on IRC and talk to them.

If I have issues with cloudflare what do I do?

Post reply on HN