Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

91–100 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#91

Post by quacksilver »

rom1v wrote: post #83
v5v3 wrote: post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

On Android, in Settings, Network & internet, Private DNS, you can only provide one in "Private DNS provider hostname" (AFAIK). Btw, I really don't understand why it does not accept an IP (1.1.1.1), so you have to give an address (one.one.one.one). It would be more sensible to configure a DNS server from an IP rather than from an address to be resolved by a DNS server :/

Private DNS on Android refers to 'DNS over HTTPS' and would normally only accept a hostname.

Normal DNS can normally be changed in your connection settings for a given connection on most flavours of Android.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#92

Post by noduerme »

jallmann wrote: post #9

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed…

Funny. I was configuring a new domain today, and for about 20 minutes I could only reach it through Firefox on one laptop. Google's DNS tools showed it active. SSH to an Amazon server that could resolve it. My local network had no idea of it. Flush cache and all. Turns out I had that one FF browser set up to use Cloudflare's DoH.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#93

Post by perlgeek »

An outage of roughly 1 hour is 0.13% of a month or 0.0114% of a year.

It would be interesting to see the service level objective (SLO) that cloudflare internally has for this service.

I've found https://www.cloudflare.com/r2-service-level-agreement/ but this seems to be for payed services, so this outage would put July in the "= 99.0%" bucket, so you'd get a 10% refund for the month if you payed for it.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#95

Post by quacksilver »

rat9988 wrote: post #76
v5v3 wrote: post #71

> For many users, not being able to resolve names using the 1.1.1.1 Resolver meant that basically all Internet services were unavailable. Don't you normally have 2 DnS servers listed on any device. So was the second also down, if not why didn't it go to that.

Not all users have configured two DNS servers?

It is highly recommended to configure two or more DNS servers incase one is down.

I would count not configuring at least two as 'user error'. Many systems require you to enter a primary and alternate server in order to save a configuration.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#96

Post by chii »

codingminds wrote: post #18
geoffpado wrote: post #6

This was quite annoying for me, having only switched my DNS server to 1.1.1.1 approximately 3 weeks ago to get around my ISP having a DNS outage. Is reasonably stable DNS really so much to ask for these days?

If you consume a service that's free of charge, it's at least not reasonable to complain if there's an outage. Like mentioned by other comments, do it on your own if you are not happy with the stability. Or just pay someone to provide it - like your ISP.. And TBH I trust my local ISP more than Google or CF. Not in availability, but it's covered by my local legislature. That's a huge difference - in a positive way.

> it's covered by my local legislature

which might not be a good thing in some jurisdictions - see the porn block in the UK (it's done via dns iirc, and trivially bypassed with a third party dns like cloudflare's).

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#97

Post by maxloh »

bauruine wrote: post #17
jallmann wrote: post #9

Good writeup. > It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address. Interesting, I was affected by this yesterday. My router (supposedly) had Cloudflare DoH enabled but nothing would resolve. Changing the DNS server to 8.8.8.8 fixed…

How does DoH work? Somehow you need to know the IP of cloudflare-dns.com first. Maybe your router uses 1.1.1.1 for this.

Yeah, your operating system will first need to resolve cloudflare-dns.com. This initial resolution will likely occur unencrypted via the network's default DNS. Only then will your system query the resolved address for its DoH requests.

Note that this introduces one query overhead per DNS request if the previous cache has expired. For this reason, I've been using https://1.1.1.1/dns-query instead.

In theory, this should eliminate that overhead. Your operating system can validate the IP address of the DNS response by using the Subject Alternative Name (SAN) field within the CA certificate presented by the DoH server: https://g.co/gemini/share/40af4514cb6e

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#98

Post by bigiain »

chrismorgan wrote: post #48
mentalgear wrote: post #40

Earlier quoted context omitted.

Its not wrong for smaller companies. But there's an argument that a big system critical company/provider like Cloudflare should be able to afford its own always on team with a night shift.

Not even a night shift, just normal working hours in another part of the world.

There are kinds big step/jumps as the size of a company goes up.

Step 1: You start out with the founders being on call 27x7x365 or people in the first 10 or 20 hires "carry the pager" on weekends and evenings and your entire company is doing unpaid rostered on call.

Step 2: You steal all the underwear.

Step 3: You have follow-the-sun office-hours support staff teams distributed around the globe with sufficient coverage for vacations and unexpected illness or resignations.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#99

Post by ripdog »

bauruine wrote: post #13
geoffpado wrote: post #6

This was quite annoying for me, having only switched my DNS server to 1.1.1.1 approximately 3 weeks ago to get around my ISP having a DNS outage. Is reasonably stable DNS really so much to ask for these days?

Why not use multiple? You can use 1.1.1.1, your ISPs and google at the same time. Or just run a resolver yourself.

>Or just run a resolver yourself.

I did this for a while, but ~300ms hangs on every DNS resolution sure do get old fast.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#100

Post by noduerme »

bigiain wrote: post #85
landgenoot wrote: post #78

Earlier quoted context omitted.

You need a domain in order to get the s in https to work

That's not correct. LetEncrypt are trialling ip address https/TLS certificates right now: https://letsencrypt.org/2025/07/01/issuing-our-first-ip-addr... They say: "In principle, there’s no reason that a certificate couldn’t be issued for an IP address rather than a domain name, and in fact the technical and policy standards for certificates have always allowed this, with a handful of certificate authorities offering…

right, this was announced about two weeks ago to some fanfare. So in principle there was no reason not to do it two decades ago? It would've been nice back then. I never heard of any certificate authority offering that.
Post reply on HN