Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

New TLS certificate for .herokuapp.com hostnames

devcenter.heroku.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: New TLS certificate for .herokuapp.com hostnames

#31

Post by jrockway »

manquer wrote: post #30
tgsovlerkhgsel wrote: post #27

Earlier quoted context omitted.

The downside is that due to a lack of serious competition, Let's Encrypt seems like an obvious choice, and thus it can be tempting to hardcode it. I have a homebrew Internet-of-shit device that I know has LE hardcoded. I'll have to take it of the wall and reflash if I switch to a new CA (or potentially when some of the changes described by tialaramex happen - I think I hardcoded the new root but I'm not 100% sure).

The acme protocol is well defined , and code is open source you could always implement your own service. Let’s encrypt only real hold is their root certificate is now in many trust stores , if you control both sides self signed certificates are perfectly fine you don’t need a CA at all

I think he's talking about the temptation to set up a pin to their root. That can break just as easily as any other pin, and of course you won't be prepared.

Re: New TLS certificate for .herokuapp.com hostnames

#32

Post by [deleted] »

Denvercoder9 wrote: post #26
tialaramex wrote: post #24

Earlier quoted context omitted.

Although, speaking of Let's Encrypt, there will be a series of disruptive events over the next 18 months or so. * Soon (although when exactly I'm not sure because it has been delayed at least once) the Let's Encrypt systems will tell compliant ACME clients that the "correct" intermediate is Let's Encrypt's ISRG-signed X3 intermediate. This is a different certificate for the same X3 private key you're used to but not…

That sounds like just one maybe-disruptive event that manifests itself differently if you keep working around it instead of dealing with it properly. If you need to deal with it at all - I suspect most systems that still need to connect to the internet trust the ISRG root nowadays.

[deleted]

Re: New TLS certificate for .herokuapp.com hostnames

#33

Post by yjftsjthsd-h »

tempay wrote: post #19
jrockway wrote: post #17

Earlier quoted context omitted.

Let's Encrypt issues wildcard certificates, so Heroku could easily pay $0 for one. I don't think they are negotiating anything.

The issue is that some of their customers have pinned DigiCert so they have two choices: * pay whatever DigiCert demands for a new certificate * accept that some of their customers will break Doing this two weeks before the old certificate expires puts them in a difficult situation for negotiating, especially now they've committed to getting a new DigiCert certificate.

It seems like the solution to this is to implement both but charge customers to use the DigiCert chain. "Oh, you went and pinned something that you shouldn't have? That's fine; you can either fix it yourself or pay us to support your mistake."

Re: New TLS certificate for .herokuapp.com hostnames

#34

Post by yjftsjthsd-h »

godzillabrennus wrote: post #20
bad_user wrote: post #16

Why would you pin a certificate that you did not generate for a domain that's not yours?

My guess is that it is some larger corporate client with a middleware app they pinned the cert into. An app they built on Heroku hurriedly because it was fast and cheap to get started and they didn’t expect to need to scale. Then as can happen it became important and they scaled anyway. They probably lost the talent that built it so they don’t even remember how it works. All of this is assumption based on how they se…

Ugh, yes, I assure you that some corporate clients will even try to pin the actual leaf certificate; pinning an intermediate or root is almost good behavior for them. (Honestly, the number of times I had to tell our support people that no, we would not support customers trying to pin our AWS-issued certificates, and no, I couldn't promise to notify them even if I wanted to since AWS could just rotate them at will...)

Re: New TLS certificate for .herokuapp.com hostnames

#35

Post by EB66 »

Denvercoder9 wrote: post #26
tialaramex wrote: post #24

Earlier quoted context omitted.

Although, speaking of Let's Encrypt, there will be a series of disruptive events over the next 18 months or so. * Soon (although when exactly I'm not sure because it has been delayed at least once) the Let's Encrypt systems will tell compliant ACME clients that the "correct" intermediate is Let's Encrypt's ISRG-signed X3 intermediate. This is a different certificate for the same X3 private key you're used to but not…

That sounds like just one maybe-disruptive event that manifests itself differently if you keep working around it instead of dealing with it properly. If you need to deal with it at all - I suspect most systems that still need to connect to the internet trust the ISRG root nowadays.

> I suspect most systems that still need to connect to the internet trust the ISRG root nowadays.

There are tons of systems that do not -- particularly in the enterprise. I manage web servers for a mission-critical healthcare-related SaaS. We occasionally encounter TLS issues even with Globalsign root certificates -- far more distributed than ISRG.

We ended up switching to DigiCert last year and it helped reduce the number of TLS-related failures reported to us.

We could never switch to Let's Encrypt / ISRG for that reason. Even if ISRG has 95% distribution of their root certificate, that's not good enough for mission-critical enterprise.

I'm not at all surprised that Heroku had to roll back their TLS certificate back to DigiCert -- DigiCert is what you want if need compatibility with the highest number of clients.

Re: New TLS certificate for .herokuapp.com hostnames

#36

Post by floatingatoll »

tialaramex wrote: post #24
jrockway wrote: post #18

Something that's nice about Let's Encrypt is that it forces you to change something every few months. After the first couple months, you'll probably get your issues worked out. If you just change certs every few years, then every few years you have some sort of disaster because of the "well we fixed it, we don't have to worry for two years" effect. A broader lesson is the importance of "trying out" rare events, even…

Although, speaking of Let's Encrypt, there will be a series of disruptive events over the next 18 months or so. * Soon (although when exactly I'm not sure because it has been delayed at least once) the Let's Encrypt systems will tell compliant ACME clients that the "correct" intermediate is Let's Encrypt's ISRG-signed X3 intermediate. This is a different certificate for the same X3 private key you're used to but not…

This inspired me to look into my system's trusted roots. Here's the root CA expirations coming up in the next 18 months. The last one on this list really hits home, as anyone who did TLS back in the early 00's may remember.

2020-09-12 - DST Root CA X4

2021-03-17 - QuoVadis Root Certification Authority

2021-04-06 - Sonera Class X2

2021-09-30 - DST Root CA X3

2021-11-09 - Admin-Root-CA

2021-12-15 - Belgium Root CA2

2021-12-15 - GlobalSign

Re: New TLS certificate for .herokuapp.com hostnames

#37

Post by mjw1007 »

tialaramex wrote: post #24
jrockway wrote: post #18

Something that's nice about Let's Encrypt is that it forces you to change something every few months. After the first couple months, you'll probably get your issues worked out. If you just change certs every few years, then every few years you have some sort of disaster because of the "well we fixed it, we don't have to worry for two years" effect. A broader lesson is the importance of "trying out" rare events, even…

Although, speaking of Let's Encrypt, there will be a series of disruptive events over the next 18 months or so. * Soon (although when exactly I'm not sure because it has been delayed at least once) the Let's Encrypt systems will tell compliant ACME clients that the "correct" intermediate is Let's Encrypt's ISRG-signed X3 intermediate. This is a different certificate for the same X3 private key you're used to but not…

I think "Six year old Android phones" is optimistic. As I understand it the ISRG root was added in Android 7.1, and 7.0 was released in August 2016.

So it might be more like "Three year old Android phones", given the lag between upstream releases and adoption.

Re: New TLS certificate for .herokuapp.com hostnames

#38

Post by tialaramex »

floatingatoll wrote: post #36
tialaramex wrote: post #24

Earlier quoted context omitted.

Although, speaking of Let's Encrypt, there will be a series of disruptive events over the next 18 months or so. * Soon (although when exactly I'm not sure because it has been delayed at least once) the Let's Encrypt systems will tell compliant ACME clients that the "correct" intermediate is Let's Encrypt's ISRG-signed X3 intermediate. This is a different certificate for the same X3 private key you're used to but not…

This inspired me to look into my system's trusted roots. Here's the root CA expirations coming up in the next 18 months. The last one on this list really hits home, as anyone who did TLS back in the early 00's may remember. 2020-09-12 - DST Root CA X4 2021-03-17 - QuoVadis Root Certification Authority 2021-04-06 - Sonera Class X2 2021-09-30 - DST Root CA X3 2021-11-09 - Admin-Root-CA 2021-12-15 - Belgium Root CA2 202…

Admin-Root-CA shows us how far we've come, I think today that even if Mozilla's root programme didn't forbid them people would guess that ultra-vague names aren't a good idea.

For reference that is the Swiss government's root and it isn't trusted by Mozilla so as a consequence it's unlikely that any systems you have facing ordinary web browsers depend on this root to be trusted.

It's also funny to go back and look at Mozilla's trust decision (it's before I was engaged in looking at this on a day-to-day basis) and see that the terrible naming was decisive while the practice of just basically trusting a Swiss government employee to issue whatever they want was considered only "problematic" and not necessarily a showstopper.

Of course because Mozilla doesn't trust this root, it does not see itself as having any oversight role for the root. So if you use MacOS, or Windows, to do anything other than run Firefox, you're reliant on their teams to verify that this root is well run. Maybe they're doing a great job? I guess you'd only ever find out the hard way because they operate entirely behind closed doors.

Re: New TLS certificate for .herokuapp.com hostnames

#39

Post by fomine3 »

mjw1007 wrote: post #37
tialaramex wrote: post #24

Earlier quoted context omitted.

Although, speaking of Let's Encrypt, there will be a series of disruptive events over the next 18 months or so. * Soon (although when exactly I'm not sure because it has been delayed at least once) the Let's Encrypt systems will tell compliant ACME clients that the "correct" intermediate is Let's Encrypt's ISRG-signed X3 intermediate. This is a different certificate for the same X3 private key you're used to but not…

I think "Six year old Android phones" is optimistic. As I understand it the ISRG root was added in Android 7.1, and 7.0 was released in August 2016. So it might be more like "Three year old Android phones", given the lag between upstream releases and adoption.

https://news.ycombinator.com/item?id=23496332

They postponed but anyway they planned to drop old Android support in 2020 but I doubt it's possible in near future.

Re: New TLS certificate for .herokuapp.com hostnames

#40

Post by photonios »

jdxcode wrote: post #28
tuna-piano wrote: post #14

Earlier quoted context omitted.

Yep, we were down during that time. I don't know what Heroku is doing with this certificate thing, but seems like a mess.

If you're pinning the SSL cert then it's not something they can fix, it's something you need to change on your end.

We went down. We're not pinning the SSL cert. We're behind CloudFront, and CloudFront didn't trust the new cert and stopped forwarding traffic.
Post reply on HN