Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Security Notification

status.heroku.com

Post reply on HN
Print view View original

111–120 of 162 posts

Re: Heroku Security Notification

#111

Post by nameless912 »

adrr wrote: post #106
lovelearning wrote: post #49

> On April 7, 2022, a threat actor obtained access...GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. Can some experienced security professionals weigh in on the cultural and organizational factors that allow this kind of major breach to go unnoticed for a week, that too in a reputed company like Heroku? I'm not asking this rhetoric…

Because Heroku/Salesforce doesn't have real security. Requiring special characters in passwords and sending out emails that have http and not https links to a password reset page. Their security is a joke.

As a former Herokai, let me color this a bit:

Heroku _used_ to have their own security team which was quite good and had some scary talented people on it. However, over the last 3 years or so Salesforce has been forcing Heroku to adopt Salesforce's operations practices, and this has not only wrecked productivity but completely destroyed morale and caused many, many of those talented people to quit. I for one decided to quit after only working there for around 8 months due to a horrific overreach by Salesforce into Heroku's operations.

Among other things, Salesforce forced us to adopt:

- their internal ticket tracking system, which _runs in an instance of salesforce_ (barf)

- their slack instance, which lost us many of our customizations and broke a bunch of integrations for weeks (I wouldn't be altogether surprised if this was one of the causes of the delay in notifying Herokai as to what was going on)

- their incident management process, which requires us to notify "Salesforce ops HQ" anytime there's an outage that meets certain criteria.

This last one was especially bad, and meant that we no longer had full agency to act during incident response situations. I had one incident I responded to where the problem got worse while we waited for Salesforce IM to spin up, so that we ended up having what would have been a 10 minute outage turn into a 2 hour outage because the issue got out of control.

In short, the problem isn't the people trying to administer Heroku; they're great folks under a lot of pressure with very few resources. The problem is, and has always been, Salesforce's "leadership" deciding what's best for a cloud platform they couldn't give less of a damn about.

Re: Heroku Security Notification

#112

Post by bluehatbrit »

beninsydney wrote: post #78
productceo wrote: post #5

What are some good Heroku alternatives these days?

Digital Ocean's "App Platform" is pretty nice, although not as slick as Heroku - https://www.digitalocean.com/products/app-platform Railway.app is pretty nice, very slick interface, this has the most "heroku-feel" - https://railway.app

I've used the DO App platform, generally it's pretty solid but getting up and running was miserable. The documentation is really thin and there's a bunch I just had to discover along the way, in particular while getting up and running with a DB as it's not clear how the certs are delivered into the environment variables etc.

It's been solid since I got up and running but took me about a day to get a simple CRUD app moved over to it.

If they can improve the documentation it'll be much better. Right now I'm not considering it for future projects though just because platforms like Fly.io and Render seem to have better docs and additional functionality that DO Apps doesn't have yet.

Re: Heroku Security Notification

#113

Post by zoomzoom »

samwillis wrote: post #68

Well, it only seems to be getting worse on this one. I’m keen to get off Heroku, but waiting for one of the newer alternatives (Render/Fly+others) to implement WAL point in time restore for Postgres. It’s the only thing keeping me on Heroku now, but is indispensable. Anyone here from them have any update on when we could see that feature made available?

You should check out tools that help bring a Heroku quality experience into your own cloud, where there are lots of database options that meet your needs.

Coherence (disclosure- I’m a cofounder) - https://www.withcoherence.com - is one option. A defined workflow for production-quality full-stack web apps with dev and production built in alongside automated test environments, including CI/CD and cloud IDEs - all configured with one high-level YAML. We’re in a very early private beta on google cloud right now - if you’re interested, please check out our site above and let us know!

Re: Heroku Security Notification

#114

Post by jeremymcanally »

VWWHFSfQ wrote: post #3

> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else?…

The bigger question to me is how did they leverage a GitHub OAuth token to gain access to an internal database unless they're storing that config in their codebase.

If that's the case...yikes.

Re: Heroku Security Notification

#115

Post by jeremyjh »

jeremymcanally wrote: post #114
VWWHFSfQ wrote: post #3

> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else?…

The bigger question to me is how did they leverage a GitHub OAuth token to gain access to an internal database unless they're storing that config in their codebase. If that's the case...yikes.

They didn’t say that happened. I’m reading it as their DB was compromised and it’s contents included GH auth tokens.

Re: Heroku Security Notification

#116

Post by t_sawyer »

xena wrote: post #103
glenngillen wrote: post #9

Earlier quoted context omitted.

Yeah, the comms around this has been very concerning. Do I need to rotate every config var on all of my apps? Re-install every add-on? While the nature of what limited things they had disclosed to date pointed to this situation part of me wanted to believe it wasn't as bad as I was assuming. And now the trendline on this suggests I should have already done everything I've outlined above. And I'm low confidence anybod…

IIRC the environment variable settings are encrypted in a physically separate database. However it may be a good idea to rotate your secrets anyways. My hunch would be that there are so many "juicy" targets on Heroku that you probably don't need to worry too much right now unless you are or work for a "juicy" target. This is gonna suck.

I posted a similar worry about the ENV's.

Why would the Github API keys not fall into the same separate database and be encrypted as well? It's especially baffling if they already have an example/process of doing this properly.

Re: Heroku Security Notification

#117

Post by martinald »

mike_d wrote: post #47
Abishek_Muthian wrote: post #41

Earlier quoted context omitted.

> but it is not a guarantee that attacker didn't move from one area to another. The incident notification seems like the customers who are using GitHub integration are the ones who are compromised, If the attacker has gained access to other accounts then it needs to be clarified so that we could take repository level mitigations as you've mentioned; Else most will just reset account passwords and be done with it.

"Access to the environment was gained by leveraging a compromised token for a Heroku machine account" This is the equivalent of saying "the car was stolen because the car keys were laying on the kitchen table." They still don't know how they got into the house to get the car keys. GitHub was just one branch that the attacker took to further access, another being the download of the accounts database. We don't know ho…

The initial update said that the Heroku internal code itself was accessed. I wonder if they grabbed that, then analysed it to find various exploits?

Re: Heroku Security Notification

#118

Post by gruez »

Mandatum wrote: post #61
lovelearning wrote: post #49

> On April 7, 2022, a threat actor obtained access...GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. Can some experienced security professionals weigh in on the cultural and organizational factors that allow this kind of major breach to go unnoticed for a week, that too in a reputed company like Heroku? I'm not asking this rhetoric…

Salesforce has been unable to attract or retain security talent. When they acquire a company, they close down the department that does security for that company - and then move everyone into the Salesforce Trust team. Unlike engineering who they typically leave alone (unless they're integrating or rebranding). In doing so, they typically lose everyone that setup the SIEM and run the SecOps center. Everything "securit…

>bad wrap

https://www.merriam-webster.com/words-at-play/usage-bad-rap-...

Re: Heroku Security Notification

#119

Post by latortuga »

cmeacham98 wrote: post #14
VWWHFSfQ wrote: post #12

Earlier quoted context omitted.

My understanding is they didn't even know about this until GitHub told them on April 13th. I'm guessing something got triggered in GitHub's system by a flurry of tokens issued to Heroku trying to enumerate private repositories. If the attacker had just played it low and slow they might never even have known at all. Who knows how long Heroku's internal systems were compromised.

Even if that's the case, it's still way after April 13th.

This has been an ongoing security incident communicated through multiple channels since the day that github announced it. I've got a dozen emails or more in my inbox, the heroku dashboard includes mention of it, and the status page includes information about it.

Re: Heroku Security Notification

#120

Post by heartbreak »

dev_tty01 wrote: post #44
cmeacham98 wrote: post #14

Earlier quoted context omitted.

Even if that's the case, it's still way after April 13th.

Heroku reported it on 4/15. Read the beginning of the string of updates on the notification page posted here. Also, https://news.ycombinator.com/item?id=31048646

And they reported that the credentials were leaked on 5/3. That took a long time.
Post reply on HN