Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloudflare 1.1.1.1 Incident on July 14, 2025

blog.cloudflare.com

Post reply on HN
Print view View original

291–300 of 391 posts

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#291

Post by notpushkin »

sophacles wrote: post #263
rvnx wrote: post #137

Earlier quoted context omitted.

Quad9 is reselling the traffic logs, so it means if you connect to secret hosts (like for your work), they will be leaked

Im sorry... what is a secret hostname that is publicly resolvable? The very idea strikes me as irresponsible and misguided.

It could be some subdomain that’s hard to guess. You can’t (generally) enumerate all subdomains through DNS, and if you use a wildcard TLS certificate (or self-signed / no cert at all), it won’t be leaked to CT logs either. Secret hostname.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#292

Post by roughly »

aftbit wrote: post #265

>Even though this release was peer-reviewed by multiple engineers I find it somewhat surprising that none of the multiple engineers who reviewed the original change in June noticed that they had added 1.1.1.0/24 to the list of prefixes that should be rerouted. I wonder what sort of human mistake or malice led to that original error. Perhaps it would be wise to add some hard-coded special-case mitigations to DLS such…

I’m generally more a “blame the tools” than “blame the people” - depending on how the system is set up and how the configs are generated, it’s easy for a change like this to slip by - especially if a bunch of the diff is autogenerated. It’s still humans doing code review, and this kind of failure indicates process problems, regardless of whether or not laziness or stupidity were also present.

But, yes, a second mitigation here would be defense in depth - in an ideal world, all your systems use the same ops/deploy/etc stack, in this one, you probably want an extra couple steps in the way of potentially taking a large public service offline.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#293

Post by freedomben »

kentonv wrote: post #282
freedomben wrote: post #279

Earlier quoted context omitted.

Cloudflare is definitely not perfect (and when they make a change that breaks the existing API contract it always makes for several miserable days for me), but on the whole Cloudflare is pretty reliable. That said, I don't use workers and don't plan to. I personally try to stay away from non cross-platform stuff because I've been burned too heavily with vendor/platform lock-in in the past.

> and when they make a change that breaks the existing API contract it always makes for several miserable days for me If we changed an API in Workers in a way that broke any Worker in production, we consider that an incident and we will roll it back ASAP. We really try to avoid this but sometimes it's hard for us to tell. Please feel free to contact us if this happens in the future (e.g. file a support ticket or file…

Thank you! To clarify it's been API contracts in the DNS record setting API that have hit me. I'm going from memory here and it's been a couple years I think so might be a bit rusty, but one example was a slight change in data type acceptance for TTL on a record. It used to take either a string or integer in the JSON but at some point started rejecting integers (or strings, whichever one I was sending at the time stopped being accepted) so the API calls were suddenly failing (to be fair that might not have technically been a violation of the contract, but it was a change in behavior that had been consistent for years and which I would not have expected). Another one was regarding returning zone_id for records where the zone_id stopped getting populated in the returned record. Luckily my code already had the zone_id because it needs that to build the URL path, but it was a rough debugging session and then I had to hack around it by either re-adding the zone ID to the returned record or removing zone ID from my equality check, neither of which were preferred solutions.

If we start using workers though I'll definitely let you know if any API changes!

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#294

Post by lxgr »

0xbadcafebee wrote: post #239
lxgr wrote: post #189

Earlier quoted context omitted.

Is that really a win in terms of latency, considering that the chance of a cache hit increases with the number of users?

Keep in mind that low latency is a different goal than reliability. If you want the lowest-latency, the anycast address of a big company will often win out, because they've spent a couple million to get those numbers. If you want most reliable, then the closest hop to you should be the most reliable (there's no accounting for poor sysadmin'ing), which is often the ISP, but sometimes not. If you run your own recursive…

> If you want most reliable, then the closest hop to you should be the most reliable (there's no accounting for poor sysadmin'ing), which is often the ISP, but sometimes not.

I'd argue that accounting for poorly managed ISP resolvers is a critical part of reasoning about reliability.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#295

Post by lxgr »

ignoramous wrote: post #270
lxgr wrote: post #230

Earlier quoted context omitted.

> DoH hosts can resolve to multiple IPs (and even different IPs for different clients)? Yes, but not from a different organization. That was GPs point with > So if you want to use DNS over HTTPS on Android, it is not possible to provide a fallback. A cross-organizational fallback is not possible with DoH in many clients, but it is with plain old DNS. > It's worth noting that DoH (DNS-over-HTTPS) traffic remained rela…

> A cross-organizational fallback is not possible with DoH in many clients, but it is with plain old DNS. That's client implementation lacking, not some issue inherent to DoH? The DoH client is configured with a URI Template, which describes how to construct the URL to use for resolution. Configuration, discovery, and updating of the URI Template is done out of band from this protocol. Note that configuration might b…

Yes, but this restriction of only a single DoH URL seems to be the norm for many popular implementations. The protocol theoretically allowing better behavior doesn't really help people using these.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#296

Post by roughly »

chrismorgan wrote: post #289
Anon1096 wrote: post #260

Earlier quoted context omitted.

I work on something at a similar scale to 1.1.1.1, if we had this kind of setup our oncall would never be asleep (well, that is almost already the case, but alas). It's easy to say "just implement X monitor and you'd have caught this" but there's a real human cost and you have to work extremely vigilently at deleting monitors or you'll be absolutely swamped with endless false positive pages. I don't think a 5 minute…

This just seems kinda fundamental: the entire service was basically down, and it took 6+ minutes to notice? I’m just increasingly perplexed at how that could be. This isn’t an advanced monitor, this is perhaps the first and most important monitor I’d expect to implement (based on no closely relevant experience).

> based on no closely relevant experience

I don’t want to devolve this to an argument from authority, but - there’s a lot of trade offs to monitoring systems, especially at that scale. Among other things, aggregation takes time at scale, and with enough metrics and numbers coming in, your variance is all over the place. A core fact about distributed systems at this scale is that something is always broken somewhere in the stack - the law of averages demands it, and so if you’re going to do an all-fire-alarm alert any time part of the system isn’t working, you’ve got alarms going off 24/7. Actually detecting that an actual incident is actually happening on a machine of the size and complexity we’re talking about within 5 minutes is absolutely fantastic.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#297

Post by roughly »

chrismorgan wrote: post #199
perlgeek wrote: post #160

Earlier quoted context omitted.

I'm sure some engineer at cloudflare is evaluating something like this right now, and try it on historical data how many false positives that would've generated in the past, if any. Thing is, it's probably still some engineering effort, and most orgs only really improve their monitoring after it turned out to be sub-optimal.

This is hardly the first 1.1.1.1 outage. It’s also probably about the first external monitoring behaviour I imagine you’d come up with. That’s why I’m surprised—more surprised the longer I think about it, actually; more than five minutes is a really long delay to notice such a fundamental breakage.

Is your external monitor working? How many checks failed, in what order? Across how many different regions or systems? Was it a transient failure? How many times do you retry, and at what cadence? Do you push your success or failure metrics? Do you pull? What if your metrics don’t make it back? How long do you wait before considering it a problem? What other checks do you run, and how long do those take? What kind of latency is acceptable for checks like that? How many false alarms are you willing to accept, and at what cadence?

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#298

Post by encom »

fragmede wrote: post #57
ben0x539 wrote: post #49

Earlier quoted context omitted.

Isn't the largest company most likely to have the DNS resolver closest to me?

Your ISP should have a DNS revolver closer to you. "Should" doesn't necessarily mean faster, however.

In case of Denmark, ISP DNS also means censored. Of course it started with CP, as it always does, then expanded to copyrights, pharmaceuticals, gambling and "terrorism". Except for the occasional Linux ISO, I don't partake in any of these topics, but I'm opposed to any kind of censorship on principle. And naturally, this doesn't stop anyone, but politicians get to stand in front of television cameras and say they're protecting children and stopping terrorists.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#299

Post by alexandrutocar »

> It’s worth noting that DoH (DNS-over-HTTPS) traffic remained relatively stable as most DoH users use the domain cloudflare-dns.com, configured manually or through their browser, to access the public DNS resolver, rather than by IP address.

I use their DNS over HTTPS and if I hadn't seen the issue being reported here, I wouldn't have caught it at all. However, this—along with a chain of past incidents (including a recent cascading service failure caused by a third-party outage)—led me to reduce my dependencies. I no longer use Cloudflare Tunnels or Cloudflare Access, replacing them with WireGuard and mTLS certificates. I still use their compute and storage, but for personal projects only.

Re: Cloudflare 1.1.1.1 Incident on July 14, 2025

#300

Post by sophacles »

croemer wrote: post #121
TheDong wrote: post #39

Earlier quoted context omitted.

I'm not surprised. Let's say you've got a metric aggregation service, and that service crashes. What does that result in? Metrics get delayed until your orchestration system redeploys that service elsewhere, which looks like a 100% drop in metrics. Most orchestration take a sec to redeploy in this case, assuming that it could be a temporary outage of the node (like a network blip of some sort). Sooo, if you alert aft…

It's not rocket science. You do a 2 stage thing: Why not check if the aggregation service has crashed before firing the alarm if it's within the first 5 minutes? How many types of false positives can there be? You just need to eliminate the most common ones and you gradually end up with fewer of them. Before you fire a quick alarm, check that the node is up, check that the service is up etc.

> How many types of false positives can there be?

Operating at the scale of cloudflare? A lot.

* traffic appears to be down 90% but we're only getting metrics from the regions of the world that are asleep because of some pipeline error

* traffic appears to be down 90% but someone put in a firewall rule causing the metrics to be dropped

* traffic appears to be down 90% but actually the counter rolled over and prometheus handled it wrong

* traffic appears to be down 90% but the timing of the new release just caused polling to show wierd numbers

* traffic appears to be down 90% but actually there was a metrics reporting spike and there was pipeline lag

* traffic appears to be down 90% but it turns out that the team that handles transit links forgot to put the right acls around snmp so we're just not collecting metrics for 90% of our traffic

* I keep getting alerts for traffic down 90%.... thousands and thousands of them, but it turns out that really its just that this rarely used alert had some bitrot and doesn't use the aggregate metrics but the per-system ones.

* traffic is actually down 90% because theres an internet routing issue (not the dns team's problem)

* traffic is actually down 90% at one datacenter because of a fiber cut somewhere

* traffic is actually down 90% because the normal usage pattern is trough traffic volume is 10% of peak traffic volume

* traffic is down 90% from 10s ago, but 10s ago there was an unusual spike in traffic.

And then you get into all sorts of additional issues caused by the scale and distributed nature of a metrics system that monitors a huge global network of datacenters.

Post reply on HN