Live data from Hacker News

Ask HN: Anonymous person sent proof of SSH access to our production server

news.ycombinator.com

101–110 of 246 posts

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#101
post #91

In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of: "Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you woul…

[deleted]

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#102
post #43

In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH. 1. firewall - only allow SSH connections from trusted static IPs 2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo,…

IMHO you can skip most of this except #2. Key-only login is very important, blocking access with fail2ban not so much. But you definitely need to send your logs to another box so you got a copy (that is unreachable for the attacker) of the logs that this guy definitely removed from the hacked system.

I do agree but a lot of it depends on the business case. Sometimes the business requirements dictate you have password authentication enabled. :(

You're right that key only logins are the easiest quick win, but it's also worth remembering that they can still be vulnerable if the box needs passphraseless keys (eg lazy automation) or if you cannot guarantee the security of those keys (eg they're generated by third parties and/or stored on third party systems). In those situations it would pay to firewall SSH and specifically whitelist the third parties IP address.

Good suggestion about the logs too. That's an often forgotten step yet invaluable when it comes to forensics.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#104
post #9

I had something similar thing happen several years ago when I was a grad student. Me and a couple friends were putting together some Facebook apps (when they were a new thing), and one guy didn't escape user input correctly. Some teenager from an Eastern European country (I can't remember which anymore) ran a script to figure out that we were vulnerable to SQL injections. He was a nice enough guy and didn't want anyt…

This is a fair point. Maybe the hacker is evil and hell bent on destroying your server / company.

But if we treat every hacker like that by default then what kind of world do we create? Certainly take prudent safety action, but then practice what many here claim to value: knowledge sharing among curious individuals.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#105

OP here. Thanks for all the responses. I took action and updated firewall settings (which were too loose), ensured that offsite backups are in place if worse comes to worst, rotated all api keys etc, meanwhile trying to contact the anonymous person. Will rebuild the servers asap as well, super glad that we have properly maintained ansible scripts. Also will try my best to convince the CEO to allocate some money for p…

This incident isn't over. You must fully scope the incident, and the only way to do that is to hire outside help. If the breach leaks and it's found that you haven't properly responded, it will destroy trust with clients and possibly expose you to legal risks.

Hire an incident response firm. They can usually react next day if you can sign a contract today.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#106

This has been covered elsewhere (like on serverfault: http://serverfault.com/a/107346/2557 ) But it comes down to: - Take existing server down immediately. I'm assuming it is not on an isolated network -- so this should really be a priority. - Prep a new patched server (with a smaller attack surface and updated security credentials) - Postmortem the old box on an isolated network. Try to understand how the attacker g…

[deleted]

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#107

OP here. Thanks for all the responses. I took action and updated firewall settings (which were too loose), ensured that offsite backups are in place if worse comes to worst, rotated all api keys etc, meanwhile trying to contact the anonymous person. Will rebuild the servers asap as well, super glad that we have properly maintained ansible scripts. Also will try my best to convince the CEO to allocate some money for p…

Hi,

If you need security professional I can help you. Is the box on AWS, if so that would be a perfect use case for us? You won't have to worry to much about costs since we're starting up we're willing to work with your budget if you provide a testimonial for our website.

Send me an email: contact@cloudhawk.io and we'll get started quickly.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#108

I agree with the individuals who say that you need to rebuild your server. You really have no idea what they've done while they were logged in or how long they had access. So take the time to be sure a replacement build is secure and then cut over. By the way, unauthorized access to a computer is a crime in every jurisdiction that I'm familiar with. It is not advisable to "test" someone else's computer and then provi…

That's true, but it's also a reason people who have done so are wise to send an anonymous proof rather than "Hi there, my name is Dale McGuyver and I hacked your server..."

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#109
Enable two factor authentication on your servers, or else two factor on a bastion server and disable external ssh other than from your bastions on all other servers.

I strongly recommend YubiKey, it is convenient and cheap and extremely secure.

With ssh it is very easy for someone to create an ssh key that does not have a passphrase. With that, it is possible to log into the server with just the key file and nothing else. At that point all it takes is someone to lose a laptop or leave a computer unlocked and unattended and someone can get access to your machines.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#110
post #98

Earlier quoted context omitted.

God forbid. No hire if they cannot reverse binary tree on the whiteboard using angular react e6.

The guy that couldn't "reverse a binary tree" came from a team that turned out to be incompetent (in operations field; they didn't see anything wrong with distributing all the packages with software using HTTP without SSL/TLS nor any cryptographic signature).

Are we talking about Homebrew guy? Was this a flaw in Homebrew or something else he worked on?
Post reply on HN