Live data from Hacker News

1.1.1.1 outage explanation

blog.cloudflare.com

41–50 of 87 posts

Re: 1.1.1.1 outage explanation

#41

Major credit to Cloudflare for publishing a clear, honest, and detailed description of what happened. I wish more companies would do this. One thing I’d be interested to know more about is why it took 17 minutes to fix. While you can and always should strive to make them less likely, outages are inevitable, so how you respond is crucial. Here the outage was very obviously caused by a deployment that I’d assume was su…

I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?

In 2000 the answer would be no. In 2018 I think it is.

Things change in a time when you would freak getting up in the morning and say google.com did not work.

Re: 1.1.1.1 outage explanation

#42
post #28
post #18

Do they just use python as pseudo code or do they actually run their attack detection in python?

Any supplementary thoughts on the fact they are running their attack detection in python? What's the point?

There is a reply in a parallel chain (timestamped after your comment) which expands on the line of thought: https://news.ycombinator.com/item?id=17204069

Re: 1.1.1.1 outage explanation

#43
post #27

Earlier quoted context omitted.

With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.

For simple PagerDuty alerts I already need 15 minutes to open the app / logs and figure out what's going on.

Good point, although the way it's described it sounds like the problem cropped up right after deploy. So they would have been watching it actively. But as said above, 17 minutes to notice, figure out what's going on, decide what to do, and propagate the resolution seems reasonable.

Re: 1.1.1.1 outage explanation

#44
post #29

Earlier quoted context omitted.

Also, Cloudflare has a huge amount of data centres by now, probably more than any other service. Even Google often underperforms them. Debatable if a few ms make a difference but it can for people living in remote areas where CF has a centre and the next 9.9.9.9/8.8.8.8 is 100ms away.

It's important to note that the cloudflare dns does not send the EDNS Client Subnet header which can have a negative impact depending on where you live.

or a positive privacy impact depending on where you live :)

Re: 1.1.1.1 outage explanation

#45
post #26
post #21

Earlier quoted context omitted.

Or 9.9.9.9 if you're not as comfortable with Google services.

For anyone wondering, 9.9.9.9 is https://www.quad9.net/ who claim to not only resolve requests but also check them against IBM X-Force's threat intelligence database.

For the curious, they also have an extremely detailed privacy policy: https://www.quad9.net/policy/

Seems like a fair deal to me. I get free DNS service, and companies sponsoring this program get metrics on threats and general Internet usage. I'm a little skeptical of their claims that individuals can't be identified from their anonymized data. E.g. I probably only get one or two hits on my personal website every week, so it might not be hard for a malicious employee to deanonymize visitors to my site.

Some highlights from the policy:

Many nations classify IP addresses as Personally-Identifiable Information (PII), and we take a conservative approach in treating IP addresses as PII in all jurisdictions in which our systems reside. Our normal course of data management does not have any IP address information or other PII logged to disk or transmitted out of the location in which the query was received. We may aggregate certain counters to larger network block levels for statistical collection purposes, but those counters do not maintain specific IP address data nor is the format or model of data stored capable of being reverse-engineered to ascertain what specific IP addresses made what queries.

There are exceptions to this storage model: In the event of events or observed behaviors which we deem malicious or anomalous, we may utilize more detailed logging to collect more specific IP address data in the process of normal network defense and mitigation. This collection and transmission off-site will be limited to IP addresses that we determine are involved in the event.

...

We do not correlate or combine information from our logs with any personal information that you have provided Quad9 for other services, or with your specific IP address.

...

Quad9 DNS Services generate and share high level anonymized aggregate statistics including threat metrics on threat type, geolocation, and if available, sector, as well as other vertical metrics including performance metrics on the Quad9 DNS Services (i.e. number of threats blocked, infrastructure uptime) when available with the Quad9 threat intelligence (TI) partners, academic researchers, or the public.

Quad9 DNS Services share anonymized data on specific domains queried (records such as domain, timestamp, geolocation, number of hits, first seen, last seen) with its threat intelligence partners. Quad9 DNS Services also builds, stores, and may share certain DNS data streams which store high level information about domain resolved, query types, result codes, and timestamp. These streams do not contain IP address information of requestor and cannot be correlated to IP address or other PII.

...

Quad9 does not track visitors over time and across third-party websites, and therefore does not respond to Do Not Track signaling.

Re: 1.1.1.1 outage explanation

#46
post #4

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…

I’m amazed at how many people abandoned other providers and blindly switched to 1.1.1.1 I can’t even use that address with the ISP Alestra in Mexico

The lack of operational knowledge is real, and there is no easy fix other than resilient defaults by a benevolent dictator (in this case, resolvers across netblock and provider demarcations) or spending the time to educate yourself first on the “why” then the “how”.

Re: 1.1.1.1 outage explanation

#47
post #26
post #21

Earlier quoted context omitted.

Or 9.9.9.9 if you're not as comfortable with Google services.

For anyone wondering, 9.9.9.9 is https://www.quad9.net/ who claim to not only resolve requests but also check them against IBM X-Force's threat intelligence database.

Also from https://www.quad9.net/faq/

Secure IP: 9.9.9.9 Provides: Security blocklist, DNSSEC, No EDNS Client-Subnet sent. If your DNS software requires a Secondary IP address, please use the secure secondary address of 149.112.112.112

Unsecured IP: 9.9.9.10 Provides: No security blocklist, DNSSEC, sends EDNS Client-Subnet. If your DNS software requires a Secondary IP address, please use the unsecured secondary address of 149.112.112.10

Client-Subnet lets providers with widely-distributed servers pick one that's near you to serve your content.

Re: 1.1.1.1 outage explanation

#48
post #27

Earlier quoted context omitted.

I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?

With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.

Did they really deploy to all 150 DCs at once? Why was this release not done in phases? Not even a canary?

Re: 1.1.1.1 outage explanation

#49

TL;DR: we should have used an IP that is not traditionally used for testing and internal stuff by everybody including Cisco.

They were given the IP block by ARPA specifically with an agreement that they would analyze the junk traffic and report back on it. No one attempting this before has been able to keep up with the amount of noise pointed at that IP. Everyone involved knew exactly what they were getting into.

Re: 1.1.1.1 outage explanation

#50
post #4

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…

I read on the Pi-Hole forums that 'fallback' is a misleading term because clients don't work that way - they will happily spread requests between two functioning DNS servers. Can anyone confirm this or provide further insight?
Post reply on HN