Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

101–110 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#101

It's crazy that both 1.1.1.1 and 1.0.0.1 where affected by the same change I guess now we should start using a completely different provider as dns backup Maybe 8.8.8.8 or 9.9.9.9

I mean, aren't we already?

My Pi-holes both use OpenDNS, Quad9, and CloudFlare for upstream.

Most of my devices use both of my Pi-holes.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#102
post #83

Earlier quoted context omitted.

On Android, in Settings, Network & internet, Private DNS, you can only provide one in "Private DNS provider hostname" (AFAIK). Btw, I really don't understand why it does not accept an IP (1.1.1.1), so you have to give an address (one.one.one.one). It would be more sensible to configure a DNS server from an IP rather than from an address to be resolved by a DNS server :/

Private DNS on Android refers to 'DNS over HTTPS' and would normally only accept a hostname. Normal DNS can normally be changed in your connection settings for a given connection on most flavours of Android.

Its DNS over TLS. Android does not support DNS over HTTPS except Google's DNS

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#103
post #39

Earlier quoted context omitted.

I'm not surprised. Let's say you've got a metric aggregation service, and that service crashes. What does that result in? Metrics get delayed until your orchestration system redeploys that service elsewhere, which looks like a 100% drop in metrics. Most orchestration take a sec to redeploy in this case, assuming that it could be a temporary outage of the node (like a network blip of some sort). Sooo, if you alert aft…

Its not wrong for smaller companies. But there's an argument that a big system critical company/provider like Cloudflare should be able to afford its own always on team with a night shift.

I think it is reasonable if the alarm trigger time is, say 5-10% of the time required to fix most problems.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#104
post #89
post #67

Earlier quoted context omitted.

If you carry on reading, its quite obvious they misconfigured a service and routed production traffic to that instead of the correct service, and the system used to do that was built in 2018 and is considered legacy (probably because you can easily deploy bad configs). Given that, I wouldn't say the summary is "inscrutable corporatese" whatever that is.

I agree it's not "inscrutable corporatese" It's carefully written so my boss's boss thinks he understands it, and that we cannot possibly have that problem because we obviously don't have any "legacy components" because we are "modern and progressive". It is, in my opinion, closer to "intentionally misleading corporatese".

Joe Shmo committed the wrong config file to production. Innocent mistake. Sally caught it in 30 seconds. We were back up inside 2 minutes. Sent Joe to the margarita shop to recover his shattered nerves. Kid deserves a raise. Etc.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#105
post #83

Earlier quoted context omitted.

On Android, in Settings, Network & internet, Private DNS, you can only provide one in "Private DNS provider hostname" (AFAIK). Btw, I really don't understand why it does not accept an IP (1.1.1.1), so you have to give an address (one.one.one.one). It would be more sensible to configure a DNS server from an IP rather than from an address to be resolved by a DNS server :/

Private DNS on Android refers to 'DNS over HTTPS' and would normally only accept a hostname. Normal DNS can normally be changed in your connection settings for a given connection on most flavours of Android.

> Private DNS on Android refers to 'DNS over HTTPS'

Yes, sorry, I did not mention it.

So if you want to use DNS over HTTPS on Android, it is not possible to provide a fallback.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#106
post #59
post #38

Earlier quoted context omitted.

My (Unifi) router is set to automatic DoH, and I think that means it's using Cloudflare and Google. Didn't notice any disruptions so either the Cloudflare DoH kept working or it used the Google one while it was down.

Check Jallmann’s response https://news.ycombinator.com/item?id=44578490#44578917 TLDR; DoH was working

AFAICS, Jallmann just left 1 comment and it was top-level. I'm not sure what you mean by "Jallmann’s response".

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#107
post #35

Earlier quoted context omitted.

Are we meant to use a domain? I've always just used the IP.

You need a domain in order to get the s in https to work

Nope. That is not correct. https://1.1.1.1/dns-query is a perfectly valid DoH resolver address I've been using for months.

Your operating system can validate the IP address of the DNS response by using the Subject Alternative Name (SAN) field within the CA certificate presented by the DoH server: https://g.co/gemini/share/40af4514cb6e

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#108
post #83

Earlier quoted context omitted.

On Android, in Settings, Network & internet, Private DNS, you can only provide one in "Private DNS provider hostname" (AFAIK). Btw, I really don't understand why it does not accept an IP (1.1.1.1), so you have to give an address (one.one.one.one). It would be more sensible to configure a DNS server from an IP rather than from an address to be resolved by a DNS server :/

Private DNS on Android refers to 'DNS over HTTPS' and would normally only accept a hostname. Normal DNS can normally be changed in your connection settings for a given connection on most flavours of Android.

Cloudflare has valid certs for 1.1.1.1

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#109
post #98

Earlier quoted context omitted.

Not even a night shift, just normal working hours in another part of the world.

There are kinds big step/jumps as the size of a company goes up. Step 1: You start out with the founders being on call 27x7x365 or people in the first 10 or 20 hires "carry the pager" on weekends and evenings and your entire company is doing unpaid rostered on call. Step 2: You steal all the underwear. Step 3: You have follow-the-sun office-hours support staff teams distributed around the globe with sufficient covera…

I confess myself bemused by your Step 2.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#110

I’m surprised at the delay in impact detection: it took their internal health service more than five minutes to notice (or at least alert) that their main protocol’s traffic had abruptly dropped to around 10% of expected and was staying there. Without ever having been involved in monitoring at that kind of scale, I’d have pictured alarms firing for something that extreme within a minute. I’m curious for description o…

There's a constant tension between speed of detection and false positive rates. Traditional monitoring systems like Nagios and Icinga have settings where they only open events/alerts if a check failed three times in a row, because spurious failed checks are quite common. If you spam your operators with lots of alerts for monitoring checks that fix themselves, you stress the unnecessarily and create alert blindness, b…

At Cloudflare’s scale on 1.1.1.1, I’d imagine you could do something comparatively simple like track ten-minute and ten-second rolling averages (I know, I know, I make that sound much easier and more practical than it actually would be), and if they differ by more than 50%, sound the alarm. (Maybe the exact numbers would need to be tweaked, e.g. 20 seconds or 80%, but it’s the idea.)

Were it much less than 1.1.1.1 itself, taking longer than a minute to alarm probably wouldn’t surprise me, but this is 1.1.1.1, they’re dealing with vasts amounts of probably fairly consistent traffic.

Post reply on HN