Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

1.1.1.1 outage explanation

blog.cloudflare.com

Post reply on HN
Print view View original

81–87 of 87 posts

Re: 1.1.1.1 outage explanation

#81

Post by iforgotpassword »

crtasm wrote: post #50
gcommer wrote: post #4

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…

I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?

Dnsmasq has different modes if you give it multiple upstream resolvers. By default it only queries the fastest one, which it determines by every now and then sending a query to all the servers and picking the one that replies first. You can tell it to always query all servers, or always query them in the order specified.

Re: 1.1.1.1 outage explanation

#83

Post by slededit »

teddyh wrote: post #52
drawkbox wrote: post #17

Great information on the outage. Looks like another version 2 syndrome side effect [1]. > Today, in an effort to reclaim some technical debt, we deployed new code that introduced Gatebot to Provision API. > What we did not account for, and what Provision API didn’t know about, was that 1.1.1.0/24 and 1.0.0.0/24 are special IP ranges. Frankly speaking, almost every IP range is "special" for one reason or another, sinc…

“ Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the…

Engineers realized that maintaining all those bug fixes was too expensive. So as an industry we collectively agreed everyone will do constant rewrites.

That win 95 fix? Doesn't matter because win 95 was auto upgraded to be unrecognizable (SaaS is a wonderful thing).

Low memory? Not our problem - go buy another computer. Remember programmer time is more valuable.

You can try and buck the trend but your dependencies wont and the customer doesn't care whose code the bug is in. You won't get any browny points so you might as well just save the effort.

It's a brave new world.

Re: 1.1.1.1 outage explanation

#85

Post by joveian »

alibert wrote: post #71
joveian wrote: post #70

Earlier quoted context omitted.

At least for personal use, OpenNIC is nice and many of the servers say they do not keep logs. I use the 185.121.177.177 (2a05:dfc7:5::53) anycast server and it works well. They are more likely to disappear randomly than the ones run by large companies. https://servers.opennic.org/

Am I the only one not comfortable using DNS servers running by random volunteers? Is there any "vouching" of the operators or regular checks on common domain on those OpenNIC servers?

Well, my computer runs a bunch of software written by random volunteers so personally I'm not that worried about it. I personally prefer that to the available alternatives. Yes it would be great to have monitoring (of all the dns services) and I'm not sure if anyone does that, but considering the perpetual tran wreck that is DNSSEC it doesn't really alter anything as all dns is vulnerable.

With https, whoever you end up contacting needs to cough up a valid certificate for the domain in the url. I run https everywhere to try to get that protection as often as possible. In practice there are still ways that dns tricks can cause trouble but they are not as bad as you might think and browsers are slowly pushing an https only web (I hear Chrome will soon start marking all http sites as "insecure" rather than https sites as secure). ssh has its own authentication method and I do try to verify new hosts via another secure chanel.

Speaking of not trusting companies, I am reminded that at one point I noticed that CentryLink seems to be intercepting all dns traffic no matter the intended destination, so without either a secure connection past the ISP or maybe a nonstandard port it may not matter what dns server you try to use. Hopefully all ISPs that do this do the horrible redirect of invalid domains thing so attempting an http connection to an invalid domain might show if this is the case (I found it trying some of the nonstandard domains that OpenNIC resolves).

Re: 1.1.1.1 outage explanation

#86

Post by dx034 »

Funcrusher wrote: post #80
dx034 wrote: post #29

Earlier quoted context omitted.

Also, Cloudflare has a huge amount of data centres by now, probably more than any other service. Even Google often underperforms them. Debatable if a few ms make a difference but it can for people living in remote areas where CF has a centre and the next 9.9.9.9/8.8.8.8 is 100ms away.

I'm on Google Fiber and 1.1.1.1 is often 1ms for me and googles DNS is ~8-12ms. It's amazing how fast it is.

I guess Google doesn't deploy DNS at each edge. Otherwise it's hard to explain that you often get >5ms in major cities.

Re: 1.1.1.1 outage explanation

#87

Post by dx034 »

Steltek wrote: post #48
dx034 wrote: post #27

Earlier quoted context omitted.

With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.

Did they really deploy to all 150 DCs at once? Why was this release not done in phases? Not even a canary?

Maybe they did, the strategy is probably to deploy to small DCs first. But that also means the DDOS detection wouldn't trigger. So probably something that didn't get caught during such tests.
Post reply on HN