Earlier quoted context omitted.
We're not discussing IP addresses, we are discussing whether their logs can leak your secret domain name.
Thats more clear, I get your point now. Again, though, that's not how most people would read the original comment. I've never even contemplated that I might generate some hostnames existence of which might be considered sensitive. It seems like a terrible idea to begin with, as I'm sure there are other avenues for those "secret" domains to be leaked. Perhaps name your secret VMs vm1, vm2, ..., instead of . But yeah,…
Cloudflare 1.1.1.1 Incident on July 14, 2025
281–290 of 391 posts
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#282It's no surprise that Cloudflare is having a service issue again. I use Cloudflare at work. Cloudflare has many bugs, and some technical decisions are absurd, such as the worker's cache.delete method, which only clears the cache contents in the data center where the Worker was invoked!!! https://developers.cloudflare.com/workers/runtime-apis/cache... In my experience, Cloudflare support is not helpful at all, trying…
Cloudflare is definitely not perfect (and when they make a change that breaks the existing API contract it always makes for several miserable days for me), but on the whole Cloudflare is pretty reliable. That said, I don't use workers and don't plan to. I personally try to stay away from non cross-platform stuff because I've been burned too heavily with vendor/platform lock-in in the past.
If we changed an API in Workers in a way that broke any Worker in production, we consider that an incident and we will roll it back ASAP. We really try to avoid this but sometimes it's hard for us to tell. Please feel free to contact us if this happens in the future (e.g. file a support ticket or file a bug on workerd on GitHub or complain in our Discord or email kenton@cloudflare.com).
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#283Earlier quoted context omitted.
Not a hardware issue, but a physics problem. I live in NZ. I guess the root servers are all in the US, so that's 130ms per trip minimum.
They are not all in the US.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#284Earlier quoted context omitted.
Yes, I would also highly recommend using a DNS closest to you (for those that have ISPs that don't mess around (blocking etc.) with their DNS you usually get much better response times) and multiple from different providers. If your device doesn't support proper failover use a local DNS forwarder on your router or an external one. In Switzerland I would use Init7 (isp that doesn't filter) -> quad9 (unfiltered Version…
How busy in life are you that we're concerning ourselves with nearest DNS? Are you browsing the internet like a high frequency stock trader? Seriously, in everyone's day to day, other than when these incidents happen, does someone notice a delay from resolving a domain name? I get that in theory blah blah, but we now have choices in who gets to see all of our requests and the ISP will always lose out to the other los…
If you choose a resolver that is very far, 100ms longer page loads do end add up quickly...
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#285Interesting to see that they probably lost 20% of 1.1.1.1 usage from a roughly 20 minute incident. Not sure how cloudflare keeps struggling with issues like these, this isn't the first (and probably won't be the last) time they have these 'simple', 'deprecated', 'legacy' issues occuring. 8.8.8.8+8.8.4.4 hasn't had a global(1) second of downtime for almost a decade. 1: localized issues did exist, but that's really the…
Cloudflare has a reasonable culture around incident response, but it doesn't incentivize proactive prevention.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#286This is a good time to mention that dnsmasq lets you setup several DNS servers, and can race them. The first responder wins. You won't ever notice one of the services being down: all-servers server=8.8.8.8 server=9.9.9.9 server=1.1.1.1
That sounds good in principle, but is there a more private configuration that doesnt send DNS resolutions to cloudfare, google et al. ie. avoid BigTech tracking, and not wanting DOH. dnsmasq with a list of smaller trusted DNS providers sounds perfect, as long as it is not considered bad etiquette to spam multiple DNS providers for every resolution? But where to find a trusted list of privacy focused DNS resolvers. Th…
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#287Earlier quoted context omitted.
Who? Honest question
Google is serving you ads, CF isn’t. And it’s not conspiracy theory - it was very suspicious when we did some testing on small, aware group. The traffic didn’t look like being handled anonymously at Google side
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#288Earlier quoted context omitted.
24/7 on-call is basically mandatory at any major network, which cloudflare is. Your contractual relations with other networks will require it.
I'm not convinced that the SWE crowd of HN, particularly the crowd showing up to every thread about AI 'agents' really knows what it takes to run a global network or what a NOC is. I know saying this on here runs the risk of Vint Cerf or someone like that showing up in my replies, but this is seriously getting out of hand now. Every HN thread that isn't about fawning over AI companies is devolving into armchair reddi…
(Have worked as SRE at large global platform)
I just mostly over the last few years tune out such responses and try not to engage them. The whole uninformed "Well, if it were me, I would simply not do that" kind of comment style has been pervasive on this site for longer than AI though, IMO.
Re: Cloudflare 1.1.1.1 Incident on July 14, 2025
#289Earlier quoted context omitted.
At Cloudflare’s scale on 1.1.1.1, I’d imagine you could do something comparatively simple like track ten-minute and ten-second rolling averages (I know, I know, I make that sound much easier and more practical than it actually would be), and if they differ by more than 50%, sound the alarm. (Maybe the exact numbers would need to be tweaked, e.g. 20 seconds or 80%, but it’s the idea.) Were it much less than 1.1.1.1 it…
I work on something at a similar scale to 1.1.1.1, if we had this kind of setup our oncall would never be asleep (well, that is almost already the case, but alas). It's easy to say "just implement X monitor and you'd have caught this" but there's a real human cost and you have to work extremely vigilently at deleting monitors or you'll be absolutely swamped with endless false positive pages. I don't think a 5 minute…