CA Root expired on 30 May 2020 • Hacker News Board

Re: CA Root expired on 30 May 2020

#31

Post by Operyl » Sat, May 30, 2020, 7:36 PM UTC

seibelj wrote: post #6

DataDog failed this morning because of root CA issue.[0] Was a fun Saturday morning with 5000 alarms blowing up my phone. [0] https://status.datadoghq.com/incidents/6bqpd511nj4h

Datadog failed, and our WAF provider failed at the same time too (internal services). It was .. rather confusing at it seemed like the sky was falling D: .

Re: CA Root expired on 30 May 2020

#32

Post by CarlHoerberg » Sat, May 30, 2020, 7:39 PM UTC

ric2b wrote: post #19

CloudAMPQ (managed RabbitMQ) was affected: https://status.cloudamqp.com/ Caused us some connections issues that required a restart of both our clients and the rabbitmq cluster.

yes, a bunch of older clusters were affected by this. They included an intermediate of USERTrust that was signed by AddTrust, clients that didn't check for alternate chains would then fail. We pushed the new chain (which now only includes the server cert and the Sectigo RSA cert), and dynamically reloaded the TLS listener in RabbitMQ, it should have solved it for most ppl, email support@cloudamqp.com if it didn't for you. We're sorry we didn't pushed this earlier. We were aware that the AddTrust would expire during the life time of the server certificate, but we assumed that all TLS client would find the valid chain regardless, that assumption was obviously wrong.

Re: CA Root expired on 30 May 2020

#33

Post by DavidSJ » Sat, May 30, 2020, 7:44 PM UTC

btgeekboy wrote: post #27

jeffbee wrote: post #20

Quick reminder from your friendly local SRE: never ever issue certificates that expire on weekends. Make certs expire in the middle of the afternoon on a business day wherever your operators live and work. The cert in question expires at May 30 10:48:38 2020 GMT, which smells suspiciously like a fixed time after the cert was generated, rather than at a well-chosen point in time.

If you get to the point where the exact expiration date on the certificate matters, you've already lost the game.

Defense in depth.

Re: CA Root expired on 30 May 2020

#34

Post by josephcsible » Sat, May 30, 2020, 7:46 PM UTC

admax88q wrote: post #28

Honestly, certificates should never expire or should expire daily. If certificate revocation works then its pointless to have expiring certs. Its just a mechanism for CAs to seek rent. If certificate revocation doesnt work then certs need to expire super frequently to limit potential damage if compromised. A certificate that expires in 20 years does absolutely nothing for security compared to a certificate that never…

Exactly. Certificate expiration has never really been about security. It's purely for practicality, so that CRLs won't grow without bound.

This is especially true now that we have OCSP stapling. From a security perspective, a short-lived certificate is exactly equivalent to a long-lived certificate with mandatory OCSP stapling and a short-lived OCSP response, but the latter is much more complicated.

And in this case since it's a root, it goes even further than that. Root CA's can't be revoked anyway, so if they're compromised, a software update to distrust it is required. There's really not a good reason for them to expire at all.

Re: CA Root expired on 30 May 2020

#35

Post by Ididntdothis » Sat, May 30, 2020, 7:49 PM UTC

admax88q wrote: post #28

Honestly, certificates should never expire or should expire daily. If certificate revocation works then its pointless to have expiring certs. Its just a mechanism for CAs to seek rent. If certificate revocation doesnt work then certs need to expire super frequently to limit potential damage if compromised. A certificate that expires in 20 years does absolutely nothing for security compared to a certificate that never…

I tend to agree. Seems dealing with expiration dates is just another burden without real security. If something goes wrong you have to revoke now and not wait for another year until the cert expires.

Re: CA Root expired on 30 May 2020

#36

Post by niffydroid » Sat, May 30, 2020, 7:51 PM UTC

Thankfully our uptime services spotted this earlier in the week. I'm terrible with certs, so no idea why a cert we brought this year is even using this root ca. To be honest, things like let's encrypt or cloud services which manage ssl is a great help

Re: CA Root expired on 30 May 2020

#37

Post by itsthecourier » Sat, May 30, 2020, 7:52 PM UTC

Oh god, today was an interesting day. 8am, updating thousands of servers CAs...

Re: CA Root expired on 30 May 2020

#38

Post by lmilcin » Sat, May 30, 2020, 7:53 PM UTC

jeffbee wrote: post #20

Quick reminder from your friendly local SRE: never ever issue certificates that expire on weekends. Make certs expire in the middle of the afternoon on a business day wherever your operators live and work. The cert in question expires at May 30 10:48:38 2020 GMT, which smells suspiciously like a fixed time after the cert was generated, rather than at a well-chosen point in time.

All my applications use a component that watches certs configured (everything in cert and trust store) and returns warning in telemetry from the application if any of the certificates is less than a week from expiration. This is checked periodically while the application runs.

This not only makes sure we don't miss expiration but also ensures we don't forget to configure any of the application.

We had a situation when the cert was replaced but the file was placed in incorrect path and was not actually used by the app. Having the app report on what is actually being in use is the best way to prevent this from ever happening.

Re: CA Root expired on 30 May 2020

#39

Post by vbezhenar » Sat, May 30, 2020, 7:55 PM UTC

admax88q wrote: post #28

Honestly, certificates should never expire or should expire daily. If certificate revocation works then its pointless to have expiring certs. Its just a mechanism for CAs to seek rent. If certificate revocation doesnt work then certs need to expire super frequently to limit potential damage if compromised. A certificate that expires in 20 years does absolutely nothing for security compared to a certificate that never…

To revoke a certificate you must keep a list of revoked certificates. Without expiration date that list would grow infinitely. And that list should be downloaded periodically by every entity which wants to verify certificate.

Re: CA Root expired on 30 May 2020

#40

Post by josephcsible » Sat, May 30, 2020, 7:55 PM UTC

luckylion wrote: post #3

Any predictions how much the usage of CURLOPT_SSL_VERIFYPEER, false will increase in the next 7 days?

IMO, there's a bit of a design flaw with curl here. There should be an easy flag to say "trust the particular certificate with this hash, no matter what's wrong with it", but there isn't, so people instead use the one that says "trust whatever certificate you get, no matter what's wrong with it".