Live data from Hacker News

Ask HN: Anonymous person sent proof of SSH access to our production server

news.ycombinator.com

91–100 of 246 posts

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#91
In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of:

"Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you would like to receive if, we would be happy to donate $X to your efforts."

Where $X is something you think you can easily part with, $100, $250, $500, $1000? This both primes the sender to be more generous, if they were on the fence as to whether to do something nefarious, and establishes some small trail to them (depending on method) in case of major problems with them later.

If the proof was not included in the email, I think it's much more likely you just received the opening email in a blackmail campaign. It's highly unlikely that server is even the entry point in that case, so cleaning it will not resolve the problem. It's just the sacrificial lamb for them to prove they've got leverage and let you stew, and they can contact you again after you think you've cleaned out the problem to let you know they still have access, and the only way to be rid of them is the pay them.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#92
post #56

To people focusing in on securing SSH: just because the person has SSH access doesn't mean that they got it through SSH. It's possible that they brute forced the password or whatever, but there's a ton of attack surface on a website and many ways they could have gotten access. If they got it through for example an XSS attack and got the SSH password/keys, securing SSH doesn't stop them from doing the same thing again…

I was one of the people offering advice on hardening SSH. I mentioned firewalling sshd to a subset of trusted IPs - which would still secure you against the above attack. I also suggested SSH keys should have passphrases, which would also mitigate against this attack (providing the passphrases are complex enough). And if anyone has root permissions to disable the firewall or change user login credentials, then they d…

> I mentioned firewalling sshd to a subset of trusted IPs - which would still secure you against the above attack.

No, it doesn't secure you against the above attack. If they can use an XSS to get full access to the server and get the SSH key, then hardening SSH does pretty much nothing, they just whitelist their IP and continue accessing the server. We don't know that they got SSH access via SSH. Until we understand how they gained access to the server, blocking off other means of access that might have nothing to do with how they gained access does nothing.

> I also love how you can take genuinely helpful posts - after all, it's better to harden SSH regardless of whether this specific attack initially came directly from SSH - and somehow turn those contributions into something negative. God bless internet messageboards.

Sure, hardening SSH is always a good idea, but until we actually understand how the SSH access was obtained, we don't know that it fixes the immediate problem. It absolutely is negative to give people information that persuades them their problem has been solved when it hasn't been solved.

I don't see any reason for you to take my post personally. It's not an attack on you, it's just pointing out that we need more understanding of the problem to actually solve it. Don't shoot the messenger.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#93
post #61

Earlier quoted context omitted.

The irony here is that your comment is a random HN comment from an account less than 10 days old :D In reality, I think random answers on HN (or any answer/advice anywhere) shouldn't be trusted, but rather taken with a grain of salt and think about whether the answer really helps you. >Touching the disks can expose you to the risk of being accused of tampering evidence I don't understand this. What do you say touchin…

> I don't understand this. What do you say touching the disks is? Like physical touch, or logging in and looking at the logs? Is this sarcasm? If not, hypothetical situation for you: OP works at a company that processes card information of customers. A hacker demonstrated gaining unauthorized access to production servers. Hacker pulls a db dump as well as any keys used in encryption of data (some bad practices here,…

>Is this sarcasm?

Yep. I was going for pointing out the fact that he obviously logged in so he 'touched' it and the OP is not so stupid as to wipe the whole drive when 90%~ of the comments of this post say no to. And I really doubt the 'logical' decision of anyone who manages to post to HN for advice will be to wipe the drive without getting a snapshot. And since it is a prod server, the same server has to be used unless they use AWS or some other cloud service.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#94
post #65
post #55

Earlier quoted context omitted.

Hire someone because they exploited a single vulnerability?

I've heard of much worse hiring practices than to base on a single display of competence.

True. Then again, I've heard of many better hiring practices as well.

Even odds it's a current employee, anyway.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#95
post #73

Earlier quoted context omitted.

Ask for a BTC address and send a tip.

Alternatively, ask them for their preferred payment method and don't force your FOTMcoin on them.

Bitcoin is flavor of the month now? Do we live on the same planet?

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#96
post #93

Earlier quoted context omitted.

> I don't understand this. What do you say touching the disks is? Like physical touch, or logging in and looking at the logs? Is this sarcasm? If not, hypothetical situation for you: OP works at a company that processes card information of customers. A hacker demonstrated gaining unauthorized access to production servers. Hacker pulls a db dump as well as any keys used in encryption of data (some bad practices here,…

>Is this sarcasm? Yep. I was going for pointing out the fact that he obviously logged in so he 'touched' it and the OP is not so stupid as to wipe the whole drive when 90%~ of the comments of this post say no to. And I really doubt the 'logical' decision of anyone who manages to post to HN for advice will be to wipe the drive without getting a snapshot. And since it is a prod server, the same server has to be used un…

Thanks for clarifying.

I believe the point the parent was getting at is that there could be other unintended consequences to taking relatively good advice.

Honestly, I'd even argue there is some better advice on server fault/HN than some professionals - but the difference is getting the professional has a paper trail that you can't say "well, some DBA on server fault told me!"

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#97
post #43

In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH. 1. firewall - only allow SSH connections from trusted static IPs 2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo,…

IMHO you can skip most of this except #2. Key-only login is very important, blocking access with fail2ban not so much. But you definitely need to send your logs to another box so you got a copy (that is unreachable for the attacker) of the logs that this guy definitely removed from the hacked system.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#98
post #55

Earlier quoted context omitted.

Hire someone because they exploited a single vulnerability?

God forbid. No hire if they cannot reverse binary tree on the whiteboard using angular react e6.

The guy that couldn't "reverse a binary tree" came from a team that turned out to be incompetent (in operations field; they didn't see anything wrong with distributing all the packages with software using HTTP without SSL/TLS nor any cryptographic signature).

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#99
I agree with the individuals who say that you need to rebuild your server. You really have no idea what they've done while they were logged in or how long they had access. So take the time to be sure a replacement build is secure and then cut over.

By the way, unauthorized access to a computer is a crime in every jurisdiction that I'm familiar with. It is not advisable to "test" someone else's computer and then provide them with proof that you accessed it in an unauthorized way.

If you communicate with the person and think that they deserve some sort of reward, that's your business. It is a nice thing to tell a neighbor that their door is unlocked -- but it's illegal to step inside the door and take a look around.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#100
post #56

Earlier quoted context omitted.

I was one of the people offering advice on hardening SSH. I mentioned firewalling sshd to a subset of trusted IPs - which would still secure you against the above attack. I also suggested SSH keys should have passphrases, which would also mitigate against this attack (providing the passphrases are complex enough). And if anyone has root permissions to disable the firewall or change user login credentials, then they d…

> I mentioned firewalling sshd to a subset of trusted IPs - which would still secure you against the above attack. No, it doesn't secure you against the above attack. If they can use an XSS to get full access to the server and get the SSH key, then hardening SSH does pretty much nothing, they just whitelist their IP and continue accessing the server. We don't know that they got SSH access via SSH. Until we understand…

> No, it doesn't secure you against the above attack. If they can use an XSS to get full access to the server and get the SSH key, then hardening SSH does pretty much nothing, they just whitelist their IP and continue accessing the server.

Someone else suggested that and my reply was that if they already have access to remotely execute code as root then they can easily gain root shell access with much less effort than having to do the above workarounds to enable the systems default OpenSSH server (and there are plentiful other ways to execute remote shells without needing SSH)

> Sure, hardening SSH is always a good idea, but until we actually understand how the SSH access was obtained, we don't know that it fixes the immediate problem.

Since you agree that hardening SSH is a good idea, then it doesn't matter how the attacker gained access to SSH, you'd recommend they review the security of their SSH server regardless. So your latter statement becomes irrelevant to the former statement.

> It absolutely is negative to give people information that persuades them their problem has been solved when it hasn't been solved.

I never once suggested this would fix their problems. In fact my language was very clear that my advice would harden against SSH attacks, specifically. However they have asked for next steps and while other people have rightfully focused on the forensics side of the investigation, I have complimented their advice with tips on hardening SSH. One recommendation doesn't have to override another :)

> I don't see any reason for you to take my post personally. It's not an attack on you, it's just pointing out that we need more understanding of the problem to actually solve it. Don't shoot the messenger.

The server is compromised thus it's already too late to "solve". However that doesn't mean people cannot offer advice on hardening against potential future attacks on new or existing infrastructure in conjunction with analyzing the point of attack on the compromised (and hopefully now isolated) equipment.

The snarky tone of my replies are because you have not offered tips that take precedence over my own recommendations, which if you had then I would have taken your criticisms seriously. But as it stands you're currently just disagreeing for the sake of disagreeing. Which is something I see far too often online and often just from people who want to seem knowledgeable but without imparting any actual knowledge and thus mitigate the risk of themselves looking stupid. Which is also why so many experienced individuals tire of contributing to public forums.

You say my advice doesn't solve the OP issues, well neither do your posts. So what was the point in posting them? I just see it as an odd kind of cyclic logic.

Post reply on HN