Live data from Hacker News

Ask HN: Anonymous person sent proof of SSH access to our production server

news.ycombinator.com

111–120 of 246 posts

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#111
post #82

Earlier quoted context omitted.

I assume he suggested Bitcoin because it's still a good way to handle money anonymously. If the person that breached the server wants to remain anonymous but get paid, that's pretty much the only way to go.

I assume he suggested asking them because Bitcoin isn't the only cryptocurrency.

I'm pretty sure the hacker won't want dogecoin or forked coin.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#113
1) Isolate that server. Treat this server as your ground zero, but assume that other systems might be compromised (including servers and devices of employees etc).

2) Rotate / delete SSH keys on all other servers, that have the same keys installed as the compromised server. Private SSH keys may have been compromised in your company. Inform all employees who had a public key on that server that their access is revoked and that their private keys may have been compromised.

3) Log all (established) SSH connections on these servers. If there are unexpected connections, handle these servers the same as the compromised one. (better: inspect SSH connections through network devices). Interrupt these connections.

4) If you cannot trust that the isolated server is the only compromised one, you should isolate the network and start investigating for more breaches. The person coming forward may have dug deeper to prevent you from shutting him out.

If there are signs for more breaches, you probably should bring experts in. Or, if you can afford it, rebuild your infrastructure on a green field. The latter one is the safest one. If you don't have complex systems and use cloud services and automation tools, this should be doable.

I hope you survive this one.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#114
post #47

Earlier quoted context omitted.

It's the same concept as auditors, there's the big 4 that you've probably heard of, and a ton of other, smaller firms with varying quality. You could go with a known firm like iSec Partners, Matasano (now NCC) or Mitnick Security. They won't be cheap - at worst they may be able to refer you to some other reputable firm if your budget is limited.

> or Mitnick Security Please no. Not Mitnick. I'd rather funnel clients towards my competitors than Kevin Mitnick. He's a skilled social engineer, and his greatest social engineering success was manipulating the media into believing he speaks for hackers in general. He is not a programmer, his opinions on cryptography aren't insighful, etc. His only skill is deception.

To be fair, are we assuming op's server wasn't compromised through deception? Smart play is usually the easiest path to a goal...

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#115
post #91

In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of: "Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you woul…

If you're going to offer a reward, I suggest making it conditional on revealing how they got in.

(This also assumes you're able to contact the person who sent the e-mail.)

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#116
post #91

In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of: "Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you woul…

[deleted]

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#117

Earlier quoted context omitted.

> I mentioned firewalling sshd to a subset of trusted IPs - which would still secure you against the above attack. No, it doesn't secure you against the above attack. If they can use an XSS to get full access to the server and get the SSH key, then hardening SSH does pretty much nothing, they just whitelist their IP and continue accessing the server. We don't know that they got SSH access via SSH. Until we understand…

> No, it doesn't secure you against the above attack. If they can use an XSS to get full access to the server and get the SSH key, then hardening SSH does pretty much nothing, they just whitelist their IP and continue accessing the server. Someone else suggested that and my reply was that if they already have access to remotely execute code as root then they can easily gain root shell access with much less effort tha…

Okay sure. They can figure out how the attacker got access and fix that, AND harden SSH. They should also audit their logs, install updates, use a linter on their JavaScript, use version control, use a library to sanitize SQL inputs, force HTTPS, do code reviews, and 100 other good development practices that have nothing to do with the problem at hand. I'm glad we agree on that.

Given that the OP doesn't know how to address their immediate problem, however, posting a bunch of random good practices is probably not very helpful.

I contributed something constructive: I recommended figuring out what the vulnerability is and fixing that over fixing random things and hoping you fix the problem by chance.

> I never once suggested this would fix their problems. In fact my language was very clear that my advice would harden against SSH attacks, specifically. However they have asked for next steps and while other people have rightfully focused on the forensics side of the investigation, I have complimented their advice with tips on hardening SSH. One recommendation doesn't have to override another :)

You said, "I was one of the people offering advice on hardening SSH. I mentioned firewalling sshd to a subset of trusted IPs - which would still secure you against the above attack."

If you want to claim you didn't say anything wrong and you were intending to suggest your solution in addition to the solution that actually solves the problem, that's your prerogative, but people can read the post history which shows that isn't true, so it would be more dignified to just admit you made a mistake. Nobody cares that you made a mistake--I'm not attacking you for that. I'm just trying to put up the correct information, since you didn't. It's not about you, so there's not much reason to take it personally, and you're not making yourself look good by claiming you didn't make mistakes that everyone can read.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#118
Whoever sent it has ego issues (common among hackers). Play on that. Act impressed with their skills, and concerned with how you may have exposed the company to attack, and ask for their advice. Email them as the sysadmin, not the CEO.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#119
post #107

OP here. Thanks for all the responses. I took action and updated firewall settings (which were too loose), ensured that offsite backups are in place if worse comes to worst, rotated all api keys etc, meanwhile trying to contact the anonymous person. Will rebuild the servers asap as well, super glad that we have properly maintained ansible scripts. Also will try my best to convince the CEO to allocate some money for p…

Hi, If you need security professional I can help you. Is the box on AWS, if so that would be a perfect use case for us? You won't have to worry to much about costs since we're starting up we're willing to work with your budget if you provide a testimonial for our website. Send me an email: contact@cloudhawk.io and we'll get started quickly.

I noticed IR services are not listed on your website.

I hate to be this guy, but you don't want to offer IR/forensic services if you don't have experience doing exactly that.

Your client can sue you if you get it wrong. (http://arstechnica.com/security/2016/01/security-firm-sued-f...)

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#120
post #91

In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of: "Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you woul…

"or, if you prefer to remain anonymous, to your chosen 501(c)3 charity."
Post reply on HN