Live data from Hacker News

Heroku CI and Review App Secrets Compromised

news.ycombinator.com

1–10 of 98 posts

Heroku CI and Review App Secrets Compromised

#1
Just got an email from Salesforce: "Action Required: Heroku security notification".

Looks like the database that stores pipeline-level config variables for both Review Apps and Heroku CI were compromised.

Per Heroku, "...any secrets you set in Review Apps and Heroku CI config may have been compromised and should be rotated".

This...is really messed up :/

Re: Heroku CI and Review App Secrets Compromised

#2
11 days ago they said "While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets."

I guess that was a lie?!

Re: Heroku CI and Review App Secrets Compromised

#3

11 days ago they said "While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets." I guess that was a lie?!

HN with the quality security advice, with all the recommendations to rotate config vars just to be safe.

Re: Heroku CI and Review App Secrets Compromised

#4
Yep, they outright lied about env vars. Incredible.

It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now.

They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situation.

If you're still on it, make your plans to move away now. Time is ticking until a major outage or another security incident like this one. See my comment history and related threads for more. Specifically this summary: https://news.ycombinator.com/item?id=31374048

Re: Heroku CI and Review App Secrets Compromised

#5
Text of the email:

At Salesforce, we understand that the confidentiality, integrity, and availability of your data is vital to your business, and we take the protection of your data very seriously. We value transparency and wanted to notify you of an issue affecting your account. Based on current progress, we plan to complete our investigation by May 30, 2022. We are continuing with remediation activities and plan to publish additional information about the incident once it’s resolved.

As reported on status.heroku.com, on April 7, 2022, a threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. On that same day, the threat actor downloaded data from another database that stores pipeline-level config vars for Review Apps and Heroku CI. This was identified on May 16, 2022, after further forensic investigation. We have no evidence of any unauthorized access to Heroku systems since April 14, 2022.

As a result, any secrets you set in Review Apps and Heroku CI config vars may have been compromised and should be rotated. In addition, any Heroku tokens stored in these pipeline config vars would potentially have allowed access to your Heroku account between April 7, 2022 and May 5, 2022, when your passwords were reset, invalidating all Heroku tokens as a result.

Please note, these pipeline-level config vars are different from standard app config vars. App config vars were not stored in this database and we have no evidence to suggest app config vars were compromised.

Re: Heroku CI and Review App Secrets Compromised

#6

Yep, they outright lied about env vars. Incredible. It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now. They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situatio…

I would like to move but there are really no good alternatives that are even close to Heroku.

Re: Heroku CI and Review App Secrets Compromised

#7

Yep, they outright lied about env vars. Incredible. It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now. They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situatio…

I would not say that they lied about the env vars. The stated line is still "env vars in apps were not compromised, but env vars in CI pipelines and review apps were". For some applications there may have been shared data in these vars - in our case (N=1) our CI pipeline and review apps had a dramatically smaller and less critical set of variables.

It still sucks that they are parceling out the information, but the claim that they outright lied is not true.

Re: Heroku CI and Review App Secrets Compromised

#8

Yep, they outright lied about env vars. Incredible. It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now. They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situatio…

I would not say that they lied about the env vars. The stated line is still "env vars in apps were not compromised, but env vars in CI pipelines and review apps were". For some applications there may have been shared data in these vars - in our case (N=1) our CI pipeline and review apps had a dramatically smaller and less critical set of variables. It still sucks that they are parceling out the information, but the c…

The lie was:

> We also wanted to address a question regarding impact to environment variables. While we confirmed that the threat actor had access to encrypted Heroku customer secrets stored in config var, the secrets are encrypted at rest and the threat actor did not access the encryption key necessary to decrypt config var secrets.

https://status.heroku.com/incidents/2413

Nowhere in that did it clarify it was speaking of app but not pipeline env vars. They had plenty of time to author that post too. Make sure you rotate those app env vars anyways as this somehow appears to be getting worse by the week.

Re: Heroku CI and Review App Secrets Compromised

#9

Yep, they outright lied about env vars. Incredible. It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now. They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situatio…

I would like to move but there are really no good alternatives that are even close to Heroku.

Well hopefully once it's gone the competition will be able to get more market share to build quality product. Heroku has been starving the entire ecosystem for years.

I don't have experience with any other PaaS's so I can't recommend one, but what you say is what I commonly hear.

Re: Heroku CI and Review App Secrets Compromised

#10

Yep, they outright lied about env vars. Incredible. It pains me to see even occasional defenders of Heroku. They're not the company they were 10 years ago. They've been gutted and left for dead years ago but the product was so good nobody noticed until now. They're not to be trusted as your platform. They simply don't have anywhere close to the manpower required to run such a platform. This was a when not if situatio…

I would like to move but there are really no good alternatives that are even close to Heroku.

Why not fly.io?
Post reply on HN