Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MongoHQ Security breach

security.mongohq.com

Post reply on HN
Print view View original

81–90 of 130 posts

Re: MongoHQ Security breach

#81

Post by sprice »

flyt wrote: post #3

Especially bad for people that were using MongoHQ to manage their DB backups to S3: As a precaution, we took additional steps on behalf of our customers to invalidate the Amazon Web Services credentials we were storing for you (for the purposes of backups to S3). While this prevents the abuse of your AWS credentials by any malicious party, it may have resulted in additional unintended consequences for your AWS enviro…

So this is why our app broke yesterday. We've spent the last 24 hours wondering how our AWS key was removed. Thankfully I was able to learn about this via HN. Still waiting on a note from MongoHQ. And when we went to file an issue with AWS there was no Premium Support option.

Re: MongoHQ Security breach

#82

Post by brandoncarl »

brandoncarl wrote: post #47

It would seem to me that New Relic license keys would also be available to the hacker(s), as they are required to link MongoHQ to New Relic. In speaking with New Relic, these keys cannot be regenerated. Wondering if anybody could advise as to what security issues that might create? I presume it could potentially give access to some infrastructure information. Have emailed their support but waiting on a reply. I'd lik…

From New Relic support:

With the license key someone could configure an application to report to the account affiliated with the key. They would not be able to access the account or use the key to retrieve any personal information from the account.

Re: MongoHQ Security breach

#83

Post by danial »

sprice wrote: post #81
flyt wrote: post #3

Especially bad for people that were using MongoHQ to manage their DB backups to S3: As a precaution, we took additional steps on behalf of our customers to invalidate the Amazon Web Services credentials we were storing for you (for the purposes of backups to S3). While this prevents the abuse of your AWS credentials by any malicious party, it may have resulted in additional unintended consequences for your AWS enviro…

So this is why our app broke yesterday. We've spent the last 24 hours wondering how our AWS key was removed. Thankfully I was able to learn about this via HN. Still waiting on a note from MongoHQ. And when we went to file an issue with AWS there was no Premium Support option.

I am commenting without knowing the specifics of your application, so apologies if it doesn't apply.

You should look into using separate AWS keys for your DB backups and whatever it is your app uses those credentials for. This not only prevents any future availability issues because of key revocation, it also allows you to set fine grained permissions on your access keys limited just to what they're being used for.

Re: MongoHQ Security breach

#84

Post by danial »

muns wrote: post #66
joshchaney wrote: post #38

Earlier quoted context omitted.

I was one of those people bitten by this last night. My client called and told me he was getting access denied when trying to upload files through his CMS. After some digging I found the S3 key had been revoked. This was concerning, as I hadn't touched the CMS code I wrote in like 3 years and I've had issues deploying old stuff to Heroku in the past. I really wish MongoHQ had contacted me first about revoking the key…

Best practice would be to create an IAM user for each purpose rather than sharing the same AWS key across all of your apps, for this exact reason

Yes, for this reason, and because having separate keys allows you set appropriate access controls limited to the function they are being used for.

Re: MongoHQ Security breach

#85

Post by mrkurt »

sprice wrote: post #81
flyt wrote: post #3

Especially bad for people that were using MongoHQ to manage their DB backups to S3: As a precaution, we took additional steps on behalf of our customers to invalidate the Amazon Web Services credentials we were storing for you (for the purposes of backups to S3). While this prevents the abuse of your AWS credentials by any malicious party, it may have resulted in additional unintended consequences for your AWS enviro…

So this is why our app broke yesterday. We've spent the last 24 hours wondering how our AWS key was removed. Thankfully I was able to learn about this via HN. Still waiting on a note from MongoHQ. And when we went to file an issue with AWS there was no Premium Support option.

We're very sorry about this, we went into a little bit of a panic when we realized that we held IAM credentials that gave full access to peoples' EC2 accounts and did what we thought was best. In hindsight, we should have gotten ahold of Amazon immediately and let them manage that process.

You should have a support issue open in your AWS portal now, if you need any help getting new keys for other apps. If you can't find it hit us up at support@mongohq.com and we'll escalate.

Re: MongoHQ Security breach

#86

Post by mrkurt »

joshchaney wrote: post #38
flyt wrote: post #3

Especially bad for people that were using MongoHQ to manage their DB backups to S3: As a precaution, we took additional steps on behalf of our customers to invalidate the Amazon Web Services credentials we were storing for you (for the purposes of backups to S3). While this prevents the abuse of your AWS credentials by any malicious party, it may have resulted in additional unintended consequences for your AWS enviro…

I was one of those people bitten by this last night. My client called and told me he was getting access denied when trying to upload files through his CMS. After some digging I found the S3 key had been revoked. This was concerning, as I hadn't touched the CMS code I wrote in like 3 years and I've had issues deploying old stuff to Heroku in the past. I really wish MongoHQ had contacted me first about revoking the key…

We're very sorry about this, we went into a little bit of a panic when we realized that we held IAM credentials that gave full access to peoples' EC2 accounts and did what we thought was best. In hindsight, we should have gotten ahold of Amazon immediately and let them manage that process.

Re: MongoHQ Security breach

#87

Post by joshchaney »

muns wrote: post #66
joshchaney wrote: post #38

Earlier quoted context omitted.

I was one of those people bitten by this last night. My client called and told me he was getting access denied when trying to upload files through his CMS. After some digging I found the S3 key had been revoked. This was concerning, as I hadn't touched the CMS code I wrote in like 3 years and I've had issues deploying old stuff to Heroku in the past. I really wish MongoHQ had contacted me first about revoking the key…

Best practice would be to create an IAM user for each purpose rather than sharing the same AWS key across all of your apps, for this exact reason

At the time this project was put together, IAM didn't exist. But I agree that this would be the best approach going forward.

Re: MongoHQ Security breach

#88

Post by sprice »

danial wrote: post #83
sprice wrote: post #81

Earlier quoted context omitted.

So this is why our app broke yesterday. We've spent the last 24 hours wondering how our AWS key was removed. Thankfully I was able to learn about this via HN. Still waiting on a note from MongoHQ. And when we went to file an issue with AWS there was no Premium Support option.

I am commenting without knowing the specifics of your application, so apologies if it doesn't apply. You should look into using separate AWS keys for your DB backups and whatever it is your app uses those credentials for. This not only prevents any future availability issues because of key revocation, it also allows you to set fine grained permissions on your access keys limited just to what they're being used for.

We're enacting these changes at the moment. Thanks!

Re: MongoHQ Security breach

#89

Post by ajsharp »

computer wrote: post #46
joelgascoigne wrote: post #42

Hi everyone. Joel here from Buffer. Some of you might have heard about our security breach on Saturday. I just wanted to leave a quick note and clarify that this MongoHQ security breach was the method used to obtain our users' access tokens, which led to the wave of spam on Saturday. This is a key final piece of our investigation which brings it back full circle. I'm very happy that we have been able to gain the full…

Have you considered not outsourcing critical parts of your company to startups that are iterating fast and potentially breaking things? Could you share the reasoning behind Buffer choosing to store their customers' private data with MongoHQ, instead of your own secure infrastructure?

As a founder of another startup (octocall.com) choosing to host our database with another company (postgres, hosted with Heroku) the decision is a simple one: You trade infrastructure management complexity for a monthly fee. Especially when you're starting out, the team is small, and time is much more expensive than whatever your database provider is charging you per month.

It's a great trade-off for most early-stage companies, because managing databases is hard. I'd rather leave it to the experts who specialize only in managing databases. You and your product team have a thousand and one other things to think about other than managing your database. Your provider may end up making mistakes, but that's part of the risk you take.

Security breaches are a mess for everyone involved, and we're in relatively new territory here in the Infrastructure As A Service space. Overall, I have little doubt that IAAS overall is a good thing. As an industry, we'll learn and improve on how to deal with all things "security", but we're clearly not there yet.

Re: MongoHQ Security breach

#90

Post by sprice »

mrkurt wrote: post #85
sprice wrote: post #81

Earlier quoted context omitted.

So this is why our app broke yesterday. We've spent the last 24 hours wondering how our AWS key was removed. Thankfully I was able to learn about this via HN. Still waiting on a note from MongoHQ. And when we went to file an issue with AWS there was no Premium Support option.

We're very sorry about this, we went into a little bit of a panic when we realized that we held IAM credentials that gave full access to peoples' EC2 accounts and did what we thought was best. In hindsight, we should have gotten ahold of Amazon immediately and let them manage that process. You should have a support issue open in your AWS portal now, if you need any help getting new keys for other apps. If you can't f…

Thanks. I appreciate being overly cautious when it comes to security rather than under cautious. And this has made us realize that we should have a unique profile for each service.
Post reply on HN