[deleted]
MongoHQ Security breach
41–50 of 130 posts
Re: MongoHQ Security breach
#42Hi everyone. Joel here from Buffer.
Some of you might have heard about our security breach on Saturday. I just wanted to leave a quick note and clarify that this MongoHQ security breach was the method used to obtain our users' access tokens, which led to the wave of spam on Saturday.
This is a key final piece of our investigation which brings it back full circle. I'm very happy that we have been able to gain the full understanding and can be confident there is no backdoor.
I want to be clear that this is still our fault. If access tokens were encrypted (which they are now) then this would have been avoided. In addition, MongoHQ have provided great insights and have much more logging in place than we have ourselves. We’re also increasing logging significantly as a result.
I've updated our security breach blog post with this information. If you want to see the full set of events, take a read here: http://open.bufferapp.com/buffer-has-been-hacked-here-is-wha...
Let me know if you have any questions about this. I'll keep an eye on this thread.
Re: MongoHQ Security breach
#43Earlier quoted context omitted.
2FA and VPNs are not exclusively the only way to secure things. X.509, bastion servers, airgaps that require physical access to a secure facility etc are also valid options, dependent on your systems and their configuration.
Granted, but an airgap would make working with some internal support tool a bit cumbersome :) Bastion servers if properly firewalled might be OK for a short term solution. The concern there is if you allow unfettered ssh (for example) is someone watching for the inevitable brute-forcing that will ensue?
Mandatory SSH keys mitigates the brute forcing risk, and turns it into a nuisance. My employer presently has this arrangement and has done so for a while. Bastions only get you in the door: different entrances for different environments, users keys are only propagated to the machines they need.
Re: MongoHQ Security breach
#44If I were running a company, I'd buy everyone a license of 1Password and make its use mandatory. Sure there are still plenty of attack vectors, but it's just too easy, and 1Password is such a cheap way to mitigate risk (not to mention you're doing your employees a huge favor by reducing their vulnerability outside of work).
Better yet, how about making sure your employees can only connect to your internal tools when on a properly secured VPN and not "externally" from home/coffee shop/airport?
Can you explain why this is better than allowing access using 2FA over HTTPS (with a non-crappy set of cipher choices)?
IE What does the VPN buy you, specifically, on the employee side?
(I understand entirely what it buys you on the other side of the equation, such as a smaller attack surface, i'm just trying to understand why you would think having a VPN would have made this particular case more secure)
Re: MongoHQ Security breach
#45One thing I hate seeing is startups and security. Not that mongo is bad or anything this is just a good time to say it. Build your product correctly from the start and have a security policy early. Don't leave XSS in your website, don't just hack code together and throw it on a server. As a security guy I love seeing new startups products because so often the code is so freshly written and immature that anyone it's a…
Capitalism often presents a choice between aggressively building unsafe products, or losing out to the competition. It's extremely hard to win in a fast market by being stable, secure, and correct, at least when it comes to software.
Re: MongoHQ Security breach
#46Hi everyone. Joel here from Buffer. Some of you might have heard about our security breach on Saturday. I just wanted to leave a quick note and clarify that this MongoHQ security breach was the method used to obtain our users' access tokens, which led to the wave of spam on Saturday. This is a key final piece of our investigation which brings it back full circle. I'm very happy that we have been able to gain the full…
Have you considered not outsourcing critical parts of your company to startups that are iterating fast and potentially breaking things? Could you share the reasoning behind Buffer choosing to store their customers' private data with MongoHQ, instead of your own secure infrastructure?
Re: MongoHQ Security breach
#47It would seem to me that New Relic license keys would also be available to the hacker(s), as they are required to link MongoHQ to New Relic.
In speaking with New Relic, these keys cannot be regenerated.
Wondering if anybody could advise as to what security issues that might create? I presume it could potentially give access to some infrastructure information. Have emailed their support but waiting on a reply.
I'd like to chime in that, this circumstance notwithstanding, MongoHQ has been great during the time we've been a customer of theirs.
Re: MongoHQ Security breach
#48Earlier quoted context omitted.
Better yet, how about making sure your employees can only connect to your internal tools when on a properly secured VPN and not "externally" from home/coffee shop/airport?
Can you explain why this is better than allowing access using 2FA over HTTPS (with a non-crappy set of cipher choices)? IE What does the VPN buy you, specifically, on the employee side? (I understand entirely what it buys you on the other side of the equation, such as a smaller attack surface, i'm just trying to understand why you would think having a VPN would have made this particular case more secure)
The value of a VPN over individually-secured HTTPS/TLS+2FA connections is that you can configure the VPN once, use very standard networking tools to continuously ensure that your internal services are only available over the VPN, and not have to worry about individually securing different internal services.
Another benefit is that as your internal userbase changes, you can revoke access from a single point and be reasonably assured that you've mitigated risk, which is something you only get with individually-secured services if you have a reliable directory system.
A problem with individually-secured ops/support systems is that most 3rd party code is not ready to be securely deployed Internet-facing.
Both approaches are totally workable, but the VPN approach is easier.
Re: MongoHQ Security breach
#49One thing I hate seeing is startups and security. Not that mongo is bad or anything this is just a good time to say it. Build your product correctly from the start and have a security policy early. Don't leave XSS in your website, don't just hack code together and throw it on a server. As a security guy I love seeing new startups products because so often the code is so freshly written and immature that anyone it's a…
Some times taking more time (and/or better coders) to do something 'right' vs. just hacking it together is the difference between failure and success. So from an individual startup's POV this may not be worth doing. Having enough users/success to warrant a security breach is a luxury problem when viewed from the eyes of an entrepreneur with no product. On the other hand, collectively, startups losing everyone's crede…
You're right that it is a luxury but you are wrong that it is unfair to expect startups to set a higher standard.
Startups have fewer assets than big companies. If you can't secure 1 webserver or you have a security breach with only 3 employees, there is something seriously wrong.
If you expect to one day have 10,000 servers you better believe I expect you to be able to handle 3 or 5 or 10 by yourself.
Re: MongoHQ Security breach
#50As much as the actual cause of the concern (apparently serious employee error) was preventable, I was still impressed by their detailed disclosure, the steps that have been taken to mitigate it, and actual details about what was compromised other than the generic "non-credit card personal data". I don't use MongoHQ for any projects right now (I evaluated their offering a year or so ago and opted to go with MongoLab,…
Seriously. I wish other services did security notifications with this much detail (I'm looking at you linode).