Earlier quoted context omitted.
[citation needed]
Read their TOS.
If it’s in the ToS, then it’s not true that “[they] weren't supposed to do anything with our gmail data”.
Cloudflare 1.1.1.1 Incident on July 14, 2025
321–330 of 391 posts
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#322Earlier quoted context omitted.
Cloudflare is a for-profit company in the US. Their privacy claims can't be believed. Even if we did believe them, we have no idea if rsolution data isn't taken by US TLA agencies.
Hm, what distinction are you trying to make here? OpenDNS is also an American company, acquired by Cisco (an American company) in 2015.
I don't know much about OpenDNS, but yes, I wouldn't trust Cisco to do anything that didn't somehow push money in their direction. I was just offering relevant information about Cloudflare.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#323This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1
That sounds good in principle, but is there a more private configuration that doesnt send DNS resolutions to cloudfare, google et al. ie. avoid BigTech tracking, and not wanting DOH. dnsmasq with a list of smaller trusted DNS providers sounds perfect, as long as it is not considered bad etiquette to spam multiple DNS providers for every resolution? But where to find a trusted list of privacy focused DNS resolvers. Th…
NextDNS. Generous free tier, very affordable paid tier. Happy customer for several years and I've never noticed an outage.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#324Earlier quoted context omitted.
How so? Does it reject a secondary DNS server that’s not in the same subnet or something similar?
It's using DNS over HTTPS, and it doesn't default the URL templates correctly when mixing (some) providers. You can set them manually though, and it works.
Ah, this is for DoH, gotcha!
This "URL template" thing seems odd – is Windows doing something like creating a URL out of the DNS IP and a pattern, e.g. 1.1.1.1 + "/foo" rel="nofollow">https:///foo" would yield https://1.1.1.1/foo?
If so, why not just allow providing an actual URL for each server?
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#325Earlier quoted context omitted.
dnsdist is AMAZINGLY easy to set up as a secure local resolver that forwards all queries to DoH (and checks SSL) and checks liveliness every second I need to do a write-up one day
Please do. I'd be curious what a secure-by-default self hosted resolver would look like.
For what it may be worth, here's a most basic (but fully working) config for running Unbound as a DoT-only forwarder:
server:
logfile: ""
log-queries: no
# adjust as necessary
interface: 127.0.0.1@53
access-control: 127.0.0.0/8 allow
infra-keep-probing: yes
tls-system-cert: yes
forward-zone:
name: "."
forward-tls-upstream: yes
forward-addr: 9.9.9.9@853#dns.quad9.net
forward-addr: 193.110.81.9@853#zero.dns0.eu
forward-addr: 149.112.112.112@853#dns.quad9.net
forward-addr: 185.253.5.9@853#zero.dns0.euRe: Cloudflare 1.1.1.1 Incident on July 14, 2025
#326> The way that Cloudflare manages service topologies has been refined over time and currently consist of a combination of a legacy and a strategic system that are synced.
This writing is just brilliant. Clear to technical and non-technical readers. Makes the in-progress migration sound way more exciting than it probably is!
> We are sorry for the disruption this incident caused for our customers. We are actively making these improvements to ensure improved stability moving forward and to prevent this problem from happening again.
This is about as good as you can get it from a company as serious and important as Cloudflare. Bravo to the writers and vetters for not watering this down.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#327> The way that Cloudflare manages service topologies has been refined over time and currently consist of a combination of a legacy and a strategic system that are synced. This writing is just brilliant. Clear to technical and non-technical readers. Makes the in-progress migration sound way more exciting than it probably is! > We are sorry for the disruption this incident caused for our customers. We are actively maki…
I can't tell if you are being sarcastic, but "legacy" is a term most often used by technical people whereas "strategic" is a term most often used by marketing and non-technical leadership. Mixing them together annoys both kinds of readers.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#328I’m surprised at the delay in impact detection: it took their internal health service more than five minutes to notice (or at least alert) that their main protocol’s traffic had abruptly dropped to around 10% of expected and was staying there. Without ever having been involved in monitoring at that kind of scale, I’d have pictured alarms firing for something that extreme within a minute. I’m curious for description o…
Having alarms firing within a minute just becomes a stress test for your alarm infrastructure. Is your alarm infrastructure able to get metrics and perform calculations consistently within a minute of real time?
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#329I’m surprised at the delay in impact detection: it took their internal health service more than five minutes to notice (or at least alert) that their main protocol’s traffic had abruptly dropped to around 10% of expected and was staying there. Without ever having been involved in monitoring at that kind of scale, I’d have pictured alarms firing for something that extreme within a minute. I’m curious for description o…
I'm not surprised. Let's say you've got a metric aggregation service, and that service crashes. What does that result in? Metrics get delayed until your orchestration system redeploys that service elsewhere, which looks like a 100% drop in metrics. Most orchestration take a sec to redeploy in this case, assuming that it could be a temporary outage of the node (like a network blip of some sort). Sooo, if you alert aft…
The real issue in your hypothetical scenario is a single bad metrics instance can bring the entire thing down. You could deploy multiple geographically distributed metrics aggregation services which establish the “canonical state” through a RAFT/PAXOS quorum. Then as long as a majority of metric aggregator instances are up the system will continue to work.
When you are building systems like 1.1.1.1 having an alert rollup of five minutes is not acceptable as it will hide legitimate downtime that lasts between 0 and 5 minutes.
You need to design systems which do not rely on orchestration to remediate short transient errors.
Disclosure: I work on a core SRE team for a company with over 500 million users.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#330This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1
Additionally, as long as you don't set strict-order, dnsmasq will automatically use all-servers for retries. If you were using systemd-resolved however, it retries all servers in the order they were specified, so it's important to interleave upstreams. Using the servers in the above example, and assuming IPv4 + IPv6: 1.1.1.1 2001:4860:4860::8888 9.9.9.9 2606:4700:4700::1111 8.8.8.8 2620:fe::fe 1.0.0.1 2001:4860:4860:…
wow good tip
I was handling an incident due to this outage. I ended up adding Google DNS resolvers using systemd-resolved, but I didn't think to interleave them!