Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.1.1.1 outage explanation

blog.cloudflare.com

Post reply on HN
Print view View original

51–60 of 87 posts

Re: 1.1.1.1 outage explanation

#51

Post by mjcl »

crtasm wrote: post #50
gcommer wrote: post #4

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…

I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?

Different OSes handle it differently. Windows tries the primary, waits 1 second and then starts trying secondaries.

Re: 1.1.1.1 outage explanation

#52

Post by teddyh »

drawkbox wrote: post #17

Great information on the outage. Looks like another version 2 syndrome side effect [1]. > Today, in an effort to reclaim some technical debt, we deployed new code that introduced Gatebot to Provision API. > What we did not account for, and what Provision API didn’t know about, was that 1.1.1.0/24 and 1.0.0.0/24 are special IP ranges. Frankly speaking, almost every IP range is "special" for one reason or another, sinc…

“Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95.

Each of these bugs took weeks of real-world usage before they were found.”

— Things You Should Never Do, Part I, (https://www.joelonsoftware.com/2000/04/06/things-you-should-...)

Re: 1.1.1.1 outage explanation

#53

Post by baby »

I switched to 1.1.1.1 when it was released and since I’ve had multiple issues with free wifis where they would fail to hijack my dns requests to allow me to login to their portal. I Assume this is a good thing but can someone explain to me why this is happening and what’s the state on improving these wifi portals?

PS: sorry for hijacking the thread.

Re: 1.1.1.1 outage explanation

#54

Post by TravelTechGuy »

crtasm wrote: post #50
gcommer wrote: post #4

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…

I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?

Can confirm. If I have a secondary DNS specified in my router, other than my pi-hole, it becomes useless. My guess is the router is either measuring response time, and goes with the most efficient, or otherwise round-robining the requests. Either way, requests bypass the pi-hole in such quantities that it became useless.

PS: someone here mentioned that this behavior is OS-dependent - nope, this happens on the router level, and all devices in my apartment suffer.

Re: 1.1.1.1 outage explanation

#55

Post by detaro »

TravelTechGuy wrote: post #54
crtasm wrote: post #50

Earlier quoted context omitted.

I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?

Can confirm. If I have a secondary DNS specified in my router, other than my pi-hole, it becomes useless. My guess is the router is either measuring response time, and goes with the most efficient, or otherwise round-robining the requests. Either way, requests bypass the pi-hole in such quantities that it became useless. PS: someone here mentioned that this behavior is OS-dependent - nope, this happens on the router…

Then your devices are likely using your router as a DNS resolver, which in turn talks to your pihole and the external one. And thus it depends on your router's OS what it does.

Re: 1.1.1.1 outage explanation

#56

Post by [deleted] »

saghm wrote: post #7

> Our FRP framework allows us to express this in clear and readable code. For example, this is part of the code responsible for performing DNS attack mitigation: > > def action_gk_dns(...): > [...] > > if port != 53: > return None > > if whitelisted_ip.get(ip): > return None > > if ip not in ANYCAST_IPS: > return None > > [...] What does this code sample have to do with FRP? This code seems extremely trivial and does…

[deleted]

Re: 1.1.1.1 outage explanation

#57

Post by phyzome »

blackbrokkoli wrote: post #23
obeattie wrote: post #20

Major credit to Cloudflare for publishing a clear, honest, and detailed description of what happened. I wish more companies would do this. One thing I’d be interested to know more about is why it took 17 minutes to fix. While you can and always should strive to make them less likely, outages are inevitable, so how you respond is crucial. Here the outage was very obviously caused by a deployment that I’d assume was su…

I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?

Yes, this exactly. And to expand on what happens around that:

- monitoring system picks up irregularity (smoothed over some window of time, which delays alerting)

- alert propagates to humans

- humans may take time to notice alert (even a page takes a few seconds to read)

- humans make decisions, may need to talk to other humans (all of what you said above)

- humans evaluate correct procedure, double-check it (you don't want them making the wrong "fix" and making something else worse, do you?)

- humans execute commands

- commands take time to run on large collections of computers (running them completely in parallel can cause thundering herd issues, in some cases)

Re: 1.1.1.1 outage explanation

#58

Post by cookiecaper »

crtasm wrote: post #50
gcommer wrote: post #4

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…

I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?

Most systems do not round-robin by default afaik. They just try the listed DNS servers in order (and resolv.conf will only respect the first 3 IIRC).

DNS fallback is often also misunderstood to mean "fall back if the domain is not found", but it really means "fall back if the name server fails to respond". If the domain is not found (i.e. servers returns a valid NXDOMAIN response), most resolvers do not consult any other name servers.

Round-robin, cascading name lookups, and other frequently desired functionality can be obtained through dnsmasq or similar caching/forwarding name servers.

Re: 1.1.1.1 outage explanation

#59

Post by bovermyer »

jacksmith21006 wrote: post #41
blackbrokkoli wrote: post #23

Earlier quoted context omitted.

I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?

In 2000 the answer would be no. In 2018 I think it is. Things change in a time when you would freak getting up in the morning and say google.com did not work.

Humans react, analyze, and communicate at the same speed we did in 2000. Our tools may have gotten better, but that only cuts down on part of the process.

Automated processes can only mitigate so many edge cases. Even then, humans need to be involved, and that slows things down.

Re: 1.1.1.1 outage explanation

#60

Post by jschwartzi »

TravelTechGuy wrote: post #54
crtasm wrote: post #50

Earlier quoted context omitted.

I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?

Can confirm. If I have a secondary DNS specified in my router, other than my pi-hole, it becomes useless. My guess is the router is either measuring response time, and goes with the most efficient, or otherwise round-robining the requests. Either way, requests bypass the pi-hole in such quantities that it became useless. PS: someone here mentioned that this behavior is OS-dependent - nope, this happens on the router…

It depends on how your router's DHCP server is configured. If you configure your router to pass its own IP address out as the DNS server for the local subnet then the router's behavior dictates how DNS works. If your router is passing out an external DNS in the DHCP configuration, then you'll get OS-dependent behavior.

My router uses a DNS resolver internally, and it will spread-cast to multiple DNS servers and use the quickest response it can get. It also caches using the TTL in the DNS response, and so it will serve up cached records transparently.

Post reply on HN