Earlier quoted context omitted.
I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?
With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.
1.1.1.1 outage explanation
31–40 of 87 posts
Re: 1.1.1.1 outage explanation
#32TL;DR: we should have used an IP that is not traditionally used for testing and internal stuff by everybody including Cisco.
Not even close. The system had a glitch because they were doing a major DNS resolver at all. It had nothing to do with the baggage that comes with 1.1.1.1 specifically.
Re: 1.1.1.1 outage explanation
#33Earlier quoted context omitted.
I'm not an expert, but is 17 minutes for: - shit is not working - is this an attack? - no it's us - how? - that's how - let's go back - have to get supervisor - roll back huge thing really that long?
With ~150 data centres, roll back alone probably took 5-10 minutes. Don't think 17 minutes is that long.
Re: 1.1.1.1 outage explanation
#34Do they just use python as pseudo code or do they actually run their attack detection in python?
It is python. They linked a presentation[1] and a talk about the bot. [1] https://speakerdeck.com/majek04/gatelogic-somewhat-functiona...
Re: 1.1.1.1 outage explanation
#35I wonder if it would be possible to express the idea that if a block being applied drops traffic well below expected levels, it must be a mistake?
Re: 1.1.1.1 outage explanation
#36Earlier quoted context omitted.
Yeah, but the reason why 1.1.1.1 is so fast for sites that use Cloudflare as DNS is because Cloudflare is the authoritative DNS for them. The only way you get that in a more generic sense is if a specialist DNS CDN provider started up that provided DNS services for all the existing CDNs (or they all agreed to some type of federated standard that let them share the same recursive multicast IP addresses for DNS resolut…
Also, Cloudflare has a huge amount of data centres by now, probably more than any other service. Even Google often underperforms them. Debatable if a few ms make a difference but it can for people living in remote areas where CF has a centre and the next 9.9.9.9/8.8.8.8 is 100ms away.
Re: 1.1.1.1 outage explanation
#37Earlier quoted context omitted.
It is python. They linked a presentation[1] and a talk about the bot. [1] https://speakerdeck.com/majek04/gatelogic-somewhat-functiona...
It's interesting. I would have expected them to use rather something low latency/high performance like c++ or erlang given their scale and performance criticality.
It has got nothing to do with packet filtering per se.
Re: 1.1.1.1 outage explanation
#38This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…
Re: 1.1.1.1 outage explanation
#39They launch these DNS services to much hype and hoopla and 3 weeks later they're down. I'm not surprised at all.
Why does anyone throw their faith behind these gigantic corporations with bad track records? Just because they're large? Because they have a lot of money? This trend towards foolishness is bizarre to me. Recognizing these people (cloudflares, googles, facebooks, so on and so forth) for what they are is a necessary trait for survival.
Re: 1.1.1.1 outage explanation
#40This is a great write up. It's also why the DNS root servers have a policy of surviving DDoS through massively over-provisioned, multi-org, anycasted redundancy rather this sort of smart DDoS mitigation that drops traffic: DNS is so critical that any risk of dropping real traffic is unacceptable. (obviously, such a scale is impractical for 99% of services) A good takeaway from this outage for the average user would b…
I can’t even use that address with the ISP Alestra in Mexico