Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DNSSEC disruption affecting .de domains – Resolved

status.denic.de

Post reply on HN
Print view View original

301–310 of 440 posts

Re: DNSSEC disruption affecting .de domains – Resolved

#301

Post by Woodi »

qazwsxedchac wrote: post #111
krystofbe wrote: post #7

Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE: RRSIG with malformed signature found for a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834) dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefor…

So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat. Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem.

> So a single configuration mistake in a single place wiped out external reachability of a major economy.

Real world beats sci-fi :) And isn't it why we love IT ? And hate it too, because of "peoples in charge"...

Re: DNSSEC disruption affecting .de domains – Resolved

#302

Post by htgb »

__float wrote: post #294
tharkun__ wrote: post #241

Earlier quoted context omitted.

No it doesn't. DNS breaks as soon as TTLs run out. It's your choice to set them so low that stuff breaks immediately.

What do you recommend then? DNS doesn't usually change that often, but if you mess it up when it does, you're in for some pain if TTLs are high!

Not the one you're replying to, but I'd keep TTL high normally and lower it one TTL ahead of a planned change.

Re: DNSSEC disruption affecting .de domains – Resolved

#304

Post by otabdeveloper4 »

qazwsxedchac wrote: post #111
krystofbe wrote: post #7

Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE: RRSIG with malformed signature found for a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834) dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefor…

So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat. Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem.

> The internet's famous "routing around damage"

...is only for Pentagon networks and military stuff. It's not for us normal people. (We get Cloudflare and FAANG bullshit instead.)

Re: DNSSEC disruption affecting .de domains – Resolved

#305

Post by thayne »

tptacek wrote: post #275
thayne wrote: post #263

Earlier quoted context omitted.

Probably the most common reason to use DNSSEC is to check a box on a list of compliance rules. And I don't think this will change anything for people who need DNSSEC for compliance.

There's no commercial compliance regime that requires DNSSEC (FedRAMP might be the only exception --- I'm uncertain about the current state of FedRAMP DNSSEC rules --- but that makes sense given that DNSSEC is a giant key escrow scheme.)

FedRAMP requires it, although like many requirements, you may be able to get out of it if you have a good reason and/or your sponsoring agency doesn't care about it.

There are also some large businesses that require, or strongly pressure SaaS providers to use DNSSEC. You can often contest that, but if you have DNSSEC, that's one less thing to argue about in the contract.

Re: DNSSEC disruption affecting .de domains – Resolved

#307

Post by tptacek »

thayne wrote: post #305
tptacek wrote: post #275

Earlier quoted context omitted.

There's no commercial compliance regime that requires DNSSEC (FedRAMP might be the only exception --- I'm uncertain about the current state of FedRAMP DNSSEC rules --- but that makes sense given that DNSSEC is a giant key escrow scheme.)

FedRAMP requires it, although like many requirements, you may be able to get out of it if you have a good reason and/or your sponsoring agency doesn't care about it. There are also some large businesses that require, or strongly pressure SaaS providers to use DNSSEC. You can often contest that, but if you have DNSSEC, that's one less thing to argue about in the contract.

Which businesses are those? (I ask because if they're North American, I have a pretty good sense of which large North American businesses even have DNSSEC signatures set up, and it's not many; small enough that you can easily memorize the list.)

Re: DNSSEC disruption affecting .de domains – Resolved

#308

Post by number6 »

qazwsxedchac wrote: post #111
krystofbe wrote: post #7

Looks like a DNSSEC issue, not a nameserver outage. Validating resolvers SERVFAIL on every .de name with EDE: RRSIG with malformed signature found for a0d5d1p51kijsevll74k523htmq406bk.de/nsec3 (keytag=33834) dig +cd amazon.de @8.8.8.8 works, dig amazon.de @a.nic.de works. Zone data is intact, DENIC just published an RRSIG over an NSEC3 record that doesn't validate against ZSK 33834. Every validating resolver therefor…

So a single configuration mistake in a single place wiped out external reachability of a major economy. It happened in the evening local time and should be fixable, modulo cache TTLs, by morning. This will limit the blast radius somewhat. Still, at this level, brittle infrastructure is a political risk. The internet's famous "routing around damage" isn't quite working here. Should make for an interesting post mortem.

There is the kritis (critical infrastructure law) law, which trys to enforce some standards to make things not as brittle.

Re: DNSSEC disruption affecting .de domains – Resolved

#309

Post by fulafel »

tptacek wrote: post #199
tom1337 wrote: post #189

Cloudflare has now disabled DNSSEC validation on their 1.1.1.1 resolver: https://www.cloudflarestatus.com/incidents/vjrk8c8w37lz

Welp. I think can call it on DNSSEC now.

OTOH there was recently a DNSSEC-saved-the-day piece of news: https://incrypted.com/en/dns-attack-on-eth-limo-was-stopped/

Re: DNSSEC disruption affecting .de domains – Resolved

#310

Post by cyberax »

tptacek wrote: post #298
cyberax wrote: post #296

Earlier quoted context omitted.

I can see the periodic rotations used as a way to keep up the operational experience. This is indeed a valid reason, although it needs to be weighted against the increased risk of compromise due to the rotation procedure itself. I'm just saying that rotating the key just in case someone compromised it is not a great idea. Doubly so if it's done infrequently enough for the operational experience to atrophy between rot…

I'm glad we agree about DNSSEC, but the rationale I'm giving you for key rotation is the same reason we use short-lived secrets everywhere in modern cryptosystems. It's not controversial (except among Unix systems administrators).

Oh, I never disagreed about the state of DNSSEC. It's horrible. Along with the rest of the DNS infrastructure (I just had the reason to remember the DNS haiku again today, unrelated to .de). My disagreement is that I believe that DNSSEC should be fixed, rather than abandoned. And I believe that this does not actually require all that much work.

And I just don't fully buy this rationale for asymmetric key rotation. It makes total sense for symmetric secrets (except for passwords).

Post reply on HN