https://status.denic.de/ says "Partial Service Disruption" for DNS Nameservice now. EDIT: it says "Service Disruption" now
"All Systems Operational"
Yes, it's fixed.
DNSSEC disruption affecting .de domains – Resolved
231–240 of 440 posts
Re: DNSSEC disruption affecting .de domains – Resolved
#232quad9 seems to be having problems with DNSSEC as well
Re: DNSSEC disruption affecting .de domains – Resolved
#233Re: DNSSEC disruption affecting .de domains – Resolved
#234Earlier quoted context omitted.
It is indeed a bit sad that Cloudflare had to turn off DNSSEC completely. But I completely understand that they don't have a production-ready, tested path to override DNSSEC validation for only some domains.
Sorry! status message was not clear. DNSSEC validation is temporarily disabled only for .de domains.
That's not much better!
Re: DNSSEC disruption affecting .de domains – Resolved
#235Earlier quoted context omitted.
Interesting "bus problem" to have in a scenario where everyone who is qualified, experienced and trusted enough to commit lives changes (or perform a revert, undo results of a botched maintenance, etc) in an emergency situation is not completely sober.
Sobriety is just factor to be weighed in an emergency situation. 30 years ago I was at a ski resort with about 50 friends having a drinking competition in the resort's main bar. Late that night two ski lodges collapsed, trapping people inside. Around midnight, soon after the winner was announced, the police entered and asked "who's able to drive a crane truck?" The winner of the competition put his hand up and inform…
Sounds like Australian police. I remember 15 or so years ago being in a big team assisting the Australian police with something on a remote farm. There were 20 people that needed to be taken back to base and one 10 seater car. Someone asked the police if everyone could get in the car and policeman shrugged and said you can try. So the policeman drove a four wheel drive across farmland with 16 people stuffed into the back.
Re: DNSSEC disruption affecting .de domains – Resolved
#236Earlier quoted context omitted.
DNS is a centralization risk, yes. Somehow we've decided this is fine. DNSSEC isn't the only issue - your TLD's nameservers could also be offline, or censored in your country.
Not really? .com and .net are still up If Let's Encrypt goes down, half of the Internet will become inaccessible in a week.
Presumably if LetsEncrypt goes down and stays down for a week, the sites that go down are the ones that see that their CA went down and at no point in the week take the option to get certs from a different CA?
Re: DNSSEC disruption affecting .de domains – Resolved
#237Crazy. I can't remember an incident like this ever happened before and it's still not fixed? .de is probably the most important unrestricted domain after .com from an economical perspective. Millions of businesses are "down".
There's a good index of major DNSSEC outages here, https://ianix.com/pub/dnssec-outages.html
Re: DNSSEC disruption affecting .de domains – Resolved
#238Earlier quoted context omitted.
The irony is that DNS is a global and distributed system meant to be resilient. It’s the DNSSEC layer on top in this case causing problems.
denic is the single source of truth for zones under .de. The only problem with DNSSEC here is that it's complex.
A complex thing where making a mistake makes your domains drop off the internet seems like a pretty big "only problem".
Re: DNSSEC disruption affecting .de domains – Resolved
#239That postmortem should be a fun read, can't wait.
Ok children, sit down and listen, uncle Culonavirus will tell you a story:
"It all began with the decommissioning of the last nuclear power plant, ..."
Re: DNSSEC disruption affecting .de domains – Resolved
#240Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE: RRSIG with malformed signature found for a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834) dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefor…
So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat. Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem.
"The internet's famous "routing around damage" isn't quite working here."
DNS is a look up service that runs on the internet.
Internet routing of IP packets is what the internet does and that is working fine (for a given value of fine).
You remind me of someone using the term "the internet is down" that really means: "I've forgotten my wifi password".