Live data from Hacker News

1.1.1.1 outage explanation

blog.cloudflare.com

31–40 of 87 posts

Re: 1.1.1.1 outage explanation

#31
post #27

Earlier quoted context omitted.

I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?

With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.

For simple PagerDuty alerts I already need 15 minutes to open the app / logs and figure out what's going on.

Re: 1.1.1.1 outage explanation

#32

TL;DR: we should have used an IP that is not traditionally used for testing and internal stuff by everybody including Cisco.

Not even close. The system had a glitch because they were doing a major DNS resolver at all. It had nothing to do with the baggage that comes with 1.1.1.1 specifically.

However a few days ago there was a bgp based outage on cloudfare DNs want there?

Re: 1.1.1.1 outage explanation

#33
post #27

Earlier quoted context omitted.

I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?

With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.

Not to mention selectively purging all of the rules created (I assume) at every edge server in their network that were blackholing all of the traffic to the resolver. There's probably a command for it, but all in all 17 minutes seems quite a respectable turnaround time.

Re: 1.1.1.1 outage explanation

#34
post #24
post #18

Do they just use python as pseudo code or do they actually run their attack detection in python?

It is python. They linked a presentation[1] and a talk about the bot. [1] https://speakerdeck.com/majek04/gatelogic-somewhat-functiona...

It's interesting. I would have expected them to use rather something low latency/high performance like c++ or erlang given their scale and performance criticality.

Re: 1.1.1.1 outage explanation

#35
What's interesting here is that the automatic cure (DDoS protection) was worse than the disease (even if there was an attack, blocking all access to the DNS servers is potentially worse than letting them get overloaded).

I wonder if it would be possible to express the idea that if a block being applied drops traffic well below expected levels, it must be a mistake?

Re: 1.1.1.1 outage explanation

#36
post #29
post #14

Earlier quoted context omitted.

Yeah, but the reason why 1.1.1.1 is so fast for sites that use Cloudflare as DNS is because Cloudflare is the authoritative DNS for them. The only way you get that in a more generic sense is if a specialist DNS CDN provider started up that provided DNS services for all the existing CDNs (or they all agreed to some type of federated standard that let them share the same recursive multicast IP addresses for DNS resolut…

Also, Cloudflare has a huge amount of data centres by now, probably more than any other service. Even Google often underperforms them. Debatable if a few ms make a difference but it can for people living in remote areas where CF has a centre and the next 9.9.9.9/8.8.8.8 is 100ms away.

It's important to note that the cloudflare dns does not send the EDNS Client Subnet header which can have a negative impact depending on where you live.

Re: 1.1.1.1 outage explanation

#37
post #34
post #24

Earlier quoted context omitted.

It is python. They linked a presentation[1] and a talk about the bot. [1] https://speakerdeck.com/majek04/gatelogic-somewhat-functiona...

It's interesting. I would have expected them to use rather something low latency/high performance like c++ or erlang given their scale and performance criticality.

This is something that only keeps track of user settings data and consequently configures the endpoints given that data.

It has got nothing to do with packet filtering per se.

Re: 1.1.1.1 outage explanation

#38
post #4

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…

Not if you're not comfortable giving up so many of your internet activity to Google but as others have pointed out there are alternatives to spread.

Re: 1.1.1.1 outage explanation

#39
This has been a comedy of errors. People have given me a lot of grief for being down on cloudflare over the past year but trouble sure does seem to follow them around doesn't it.

They launch these DNS services to much hype and hoopla and 3 weeks later they're down. I'm not surprised at all.

Why does anyone throw their faith behind these gigantic corporations with bad track records? Just because they're large? Because they have a lot of money? This trend towards foolishness is bizarre to me. Recognizing these people (cloudflares, googles, facebooks, so on and so forth) for what they are is a necessary trait for survival.

Re: 1.1.1.1 outage explanation

#40
post #4

This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…

I’m amazed at how many people abandoned other providers and blindly switched to 1.1.1.1

I can’t even use that address with the ISP Alestra in Mexico

Post reply on HN