Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Heroku Security Notification

status.heroku.com

Post reply on HN
Print view View original

61–70 of 162 posts

Re: Heroku Security Notification

#61

Post by Mandatum »

lovelearning wrote: post #49

> On April 7, 2022, a threat actor obtained access...GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. Can some experienced security professionals weigh in on the cultural and organizational factors that allow this kind of major breach to go unnoticed for a week, that too in a reputed company like Heroku? I'm not asking this rhetoric…

Salesforce has been unable to attract or retain security talent. When they acquire a company, they close down the department that does security for that company - and then move everyone into the Salesforce Trust team. Unlike engineering who they typically leave alone (unless they're integrating or rebranding).

In doing so, they typically lose everyone that setup the SIEM and run the SecOps center. Everything "security" ends looking the same.

They don't pay well, executives have pulled talks and fired speakers who do things they disagree with (the same executives are promoted and remain there - no accountability), they've got a pretty bad wrap within the industry.

Re: Heroku Security Notification

#62

Post by photon12 »

There's going to be a question about the expected probability of this across cloud service providers.

I've done security work for multiple cloud service providers and know a lot of people in the industry. I'm not really privy to give details.

I can say: dev teams face limits on what they can build securely, platform teams face limits on what secure by default and monitoring features they have time to implement, security operations teams have a lot of data points to look at, and in theory even changes in personnel in a couple of teams can have an impact on the threat posture for a given set of a company.

People are trying. But if you do the math for introduction of attack surface over time versus risk mitigation effort over time for that attack surface you can derive some estimates for likelihood of attack.

If your cloud provider isn't providing you that data as a customer, it's not simple to make a determination about likelihood of risk introduction and breach of your cloud provider.

This hasn't really been something people talk about because there was a tacit assumption that the biggest companies are mostly getting this right.

I'm not trying to say the cloud is inherently broken, I run a lot of workloads in the cloud and trust sensitive data to the cloud. But I do wish there were better ways for customers to have data points with which they could evaluate platforms besides extrapolating the history of public breach reports.

Re: Heroku Security Notification

#64

Post by [deleted] »

stevebmark wrote: post #55
tossaway2134 wrote: post #53

This isn't the first time Salesforce Cybersecurity has left us in the lurch while they perform damage control. On 17 May 2019, Salesforce performed maintenance on their databases that clear permission sets for users. My team was able to piece together that the incident happened at about 0200 CDT, and Salesforce didn't take ANY noticeable action for at least 9 hours when they locked all customers out of the platform.…

Salesforce’s monitoring, availability, incident handling, transparency, and accountability, are abysmal. We too we’re affected by the incident. I sincerely hope they lose some of their compliance certifications, because their behavior is unacceptable.

[deleted]

Re: Heroku Security Notification

#65

Post by sofixa »

VWWHFSfQ wrote: post #3

> Separately, our investigation also revealed that the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts. What else was in this database? Typically the password field is stored alongside the rest of the user record. So was this the entire customer database that was stolen? Usernames, emails, salted/hashed passwords, what else?…

> I feel like at some point they're just going to go completely radio silent because the extent of the breach will become such that they'll have no choice but to just lawyer up

They can't, they surely have EU customers so have to follow GDPR disclosure rules, which they might already be afoul of.

Re: Heroku Security Notification

#66

Post by sofixa »

tossaway2134 wrote: post #53

This isn't the first time Salesforce Cybersecurity has left us in the lurch while they perform damage control. On 17 May 2019, Salesforce performed maintenance on their databases that clear permission sets for users. My team was able to piece together that the incident happened at about 0200 CDT, and Salesforce didn't take ANY noticeable action for at least 9 hours when they locked all customers out of the platform.…

> Nevertheless, our users lost their jobs and were stripped of their licenses

Why? Maybe I'm assuming good intentions, but a) did they know they were seeing records they shouldn't be; b) did they report that? Even with yes and no, firing for that seems a little too much.

Re: Heroku Security Notification

#67

Post by yuliyp »

cube00 wrote: post #57
lovelearning wrote: post #49

> On April 7, 2022, a threat actor obtained access...GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. Can some experienced security professionals weigh in on the cultural and organizational factors that allow this kind of major breach to go unnoticed for a week, that too in a reputed company like Heroku? I'm not asking this rhetoric…

> GitHub identified the activity on April 12, 2022, and notified Salesforce on April 13, 2022, at which time we began our investigation. As a result, on April 16, 2022, we revoked all GitHub integration OAuth tokens, preventing customers from deploying apps from GitHub through the Heroku Dashboard or via automation. The three days after being notified to actually revoke the tokens isn't ideal either. Surely if GitHub…

I think (based on what I've read on the linked page) the notice was telling Heroku that "hey, someone used a compromised OAuth token to download your source code" not that "the tokens that you are using to read Github repositories of your users are compromised". Both are Github OAuth tokens, but playing different roles. Presumably the compromise of source code might have been used to help get access to the database that had the Github integration OAuth tokens, and realizing that might indeed have taken a couple days.

Re: Heroku Security Notification

#68

Post by samwillis »

Well, it only seems to be getting worse on this one.

I’m keen to get off Heroku, but waiting for one of the newer alternatives (Render/Fly+others) to implement WAL point in time restore for Postgres. It’s the only thing keeping me on Heroku now, but is indispensable.

Anyone here from them have any update on when we could see that feature made available?

Re: Heroku Security Notification

#69

Post by pm90 »

photon12 wrote: post #62

There's going to be a question about the expected probability of this across cloud service providers. I've done security work for multiple cloud service providers and know a lot of people in the industry. I'm not really privy to give details. I can say: dev teams face limits on what they can build securely, platform teams face limits on what secure by default and monitoring features they have time to implement, secur…

Agreed. cloud providers’ incentives are aligned with growth which naturally mean easy accessibility; hence all the defaults being generally “open”. Its so easy to make a resource accessible to anyone, or an IP accessible from anywhere in the cloud without proper restrictions by internal teams in the company; and often the default is to give teams superadmin to “unblock their time sensitive project” rather than maintaining principle of least access which requires more discipline (and thus effort).

Either cloud providers need to assume more responsibility for security or a Federal Agency like the FBI or NIST need to be more proactively engaged in improving the security posture of cloud hosted US corps.

Post reply on HN