It is weird to start the day with a Salesforce email in your inbox - had to go checking if it was valid.
Heroku Security Notification
11–20 of 162 posts
Re: Heroku Security Notification
#12Why am I hearing about this on checks the fucking date on May 5th instead of, like, month ago.
My understanding is they didn't even know about this until GitHub told them on April 13th. I'm guessing something got triggered in GitHub's system by a flurry of tokens issued to Heroku trying to enumerate private repositories. If the attacker had just played it low and slow they might never even have known at all.
Who knows how long Heroku's internal systems were compromised.
Re: Heroku Security Notification
#13> Access to the environment was gained by leveraging a compromised token for a Heroku machine account.
Any idea if this involves AWS EC2 Instance Roles? It’s incredibly convenient, but has got to be the scariest feature to enable on a platform that allows arbitrary user code to execute.
Re: Heroku Security Notification
#14Why am I hearing about this on checks the fucking date on May 5th instead of, like, month ago.
My understanding is they didn't even know about this until GitHub told them on April 13th. I'm guessing something got triggered in GitHub's system by a flurry of tokens issued to Heroku trying to enumerate private repositories. If the attacker had just played it low and slow they might never even have known at all. Who knows how long Heroku's internal systems were compromised.
Even if that's the case, it's still way after April 13th.
Re: Heroku Security Notification
#15No post body was provided.
Re: Heroku Security Notification
#16> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else?…
The last couple large places I work specifically split out the auth/password DB from the rest of the user data. They're used for different things and they have different types of sensitivity around them.
Yep. Same thinking lead to the split of /etc/shadow from /etc/passwd.
Re: Heroku Security Notification
#17> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else?…
The last couple large places I work specifically split out the auth/password DB from the rest of the user data. They're used for different things and they have different types of sensitivity around them.
This is also why folks advocate for a separate auth system (keycloak was just mentioned here on HN) since they are different types of information.
>I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up.
I feel like they are heading this route as well. Possibly even withholding information in order to save the company from mass exodus due to the incident. I'm sure they'll be fine.
Re: Heroku Security Notification
#18Earlier quoted context omitted.
My understanding is they didn't even know about this until GitHub told them on April 13th. I'm guessing something got triggered in GitHub's system by a flurry of tokens issued to Heroku trying to enumerate private repositories. If the attacker had just played it low and slow they might never even have known at all. Who knows how long Heroku's internal systems were compromised.
Even if that's the case, it's still way after April 13th.
it might be that it took this amount of time to establish the facts of the events. If they recounted an incorrect version early, it might do more damage than not telling it.
I dont know if the github disclosure "includes" heroku's disclosure : https://github.blog/2022-04-15-security-alert-stolen-oauth-u... - but it was at least april 15th - close-ish to when the event occurred.
Re: Heroku Security Notification
#19What are some good Heroku alternatives these days?
I've seen fly/io and render/com mentioned in these Heroku discussions, but I have yet to test either of them.
Re: Heroku Security Notification
#20> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else?…
The last couple large places I work specifically split out the auth/password DB from the rest of the user data. They're used for different things and they have different types of sensitivity around them.
We moved basically everything but username into an entirely different db, went so far as to hash the username column so we don’t even know what’s it is until you log in.