Live data from Hacker News

Ask HN: Anonymous person sent proof of SSH access to our production server

news.ycombinator.com

41–50 of 246 posts

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#41
post #32
post #13

Give the person a small reward for pointing out the vulnerability and offer another small reward for suggesting how to fix it.

why small ? the reward should be in accordance with the criticity IMHO.

Rewards should always be in accordance with made-up words IMHO.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#42
post #20
post #12

Earlier quoted context omitted.

Or contact the person if you can and ask them how they got in maybe even offer him a financial reward. Since he contact you Anonymously and is not trying to extort you he's just trying to point the issue out so there's no point in over reacting.

How much would you trust that person? Enough to potentially risk your business on them?

It's not about trust it's about information gathering the goal is 2 fold

1. figure out the intentions of the individual

2. quickly finding and fixing the affected system

To be clear it doesn't matter if the information is true or false because if it's true you can find evidence on the system to confirm it and if it's false it could still prove useful.

You can nuke it from orbit later that could take hours or even days depending on how much stuff you have on it plus if the entry-point was trough the new app you just created nuking it won't fix the issue.

The moment you put the app on the new server you opened yourself up to get hacked again.

We all know a constantly updated system with nothing happening on it is incredibly hard to hack vs a system that has a lot of things happening on it the more things you're doing on a server the higher the attack surface plus it's a small company we're talking about here so they probably want to keep costs down by doing everything on as few servers as possible.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#43
In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH.

1. firewall - only allow SSH connections from trusted static IPs

2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo, edit /etc/ssh/sshd_config

    PasswordAuthentication no
    ChallengeResponseAuthentication no
    # restart sshd
(edit: make sure the SSH keys have passphrases. That way you have an extra bit of security in case any workstations get compromised)

3. Disable root access. edit /etc/ssh/sshd_config

    PermitRootLogin no
    # restart sshd
4. Limit SSH access to specific user accounts. This prevents users creating their own key (in the case of mountable home directories) or other machine accounts with passwords (if you've not done #2):

    groupadd sshaccess
    # add all users to the sshaccess group. lots of different ways to do this. The following will work on some flavours of Linux but not all:
    usermod -a -G sshaccess $USER
    # now edit /etc/ssh/sshd_config and add the following in (it wont already exist)
    AllowGroups sshaccess
    # restart sshd
5. Install auto-firewalling for failed SSH logins. I personally favour fail2ban as that covers other scenarios too, but I've also used denyhosts and that's worked well for SSH.

6. Lastly, and by far the best option, don't enable SSH on any internet facing IPs.

If you need SFTP enabled, then let me know and I'll post some details on how to harden SFTP so attackers cannot gain an SSH shell.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#44

Why would you have and production system exposing SSH to the public? If you must, at least do these steps: - Disable password SSH login - Install root kit scanner, like rkhunter and check if your networked systems are infected. s/he might gained access to other instances in your infra. - Use port scanning on all your instances and check if there is any suspecious rpc port is open that you are not familiar wtih. - Ena…

There's nothing inherently wrong with exposing SSH on your production servers to the Internet. It is one of the most secure services that can run on any given host. Surely it's more secure than your web server or application daemon(s) which handle the other publicly-facing functions of your production host.

If you have the infrastructure and capability to put it on a different network by all means make it inaccessible but for most businesses there's really no other option anyway.

Simpler (better, IMHO) advice would be to make key-based authentication mandatory for your production servers. That way a brute force attack is unlikely to ever succeed. It also rules out stealing passwords since the attacker would need to obtain the entire SSH key before they could login.

Having said that, we don't know how the attacker got in. They could have created an account for themselves or changed the root password/system configuration via a vulnerability. If that's the case they could modify sshd_config so that it listens on the public IP which would make "don't expose it to the public" moot (firewalls notwithstanding).

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#46
To people focusing in on securing SSH: just because the person has SSH access doesn't mean that they got it through SSH. It's possible that they brute forced the password or whatever, but there's a ton of attack surface on a website and many ways they could have gotten access. If they got it through for example an XSS attack and got the SSH password/keys, securing SSH doesn't stop them from doing the same thing again.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#47
post #8

[Edit: not-OP] As the 2 comments so-far have suggested getting security experts, where would be a good place to source security experts? I'm envisioning 2 kinds: * Consultant, working for a fee (with retainer?); * Independent, may be consultant, but could also be someone currently looking for a new permanent role and would bring welcome diversity/expertise to a small team - potentially illiquid / poorly matched hirin…

> where would be a good place to source security experts? There's no universal good answer for this. I spend a lot of time on ##crypto (irc.freenode.net), and a lot of smart folks hang out there. Some are very well connected to other security experts in their own isolated communities. However, there are undoubtedly silos of security expertise that remain untapped if you rely on just IRC. You could also find folks who…

It's the same concept as auditors, there's the big 4 that you've probably heard of, and a ton of other, smaller firms with varying quality.

You could go with a known firm like iSec Partners, Matasano (now NCC) or Mitnick Security. They won't be cheap - at worst they may be able to refer you to some other reputable firm if your budget is limited.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#48
post #43

In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH. 1. firewall - only allow SSH connections from trusted static IPs 2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo,…

You assume the breach happened over SSH. This is valuable information to securing SSH, but it's entirely possible the original breach happened over some other service, and there were some other steps involved in the breach before the SSH screenshot was taken.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#49
post #43

In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH. 1. firewall - only allow SSH connections from trusted static IPs 2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo,…

Keep in mind though that while securing SSH is a good approach, SSH itself is very unlikely to be the route of compromise unless an extremely insecure account were present with weak password auth.

It's far more likely that the attacker got legit credentials via another means, web application vulnerability, social engineering or malware attack on company machines, etc. I'd look at the less common applications you run, particularly anything that doesn't particularly look like it was designed to run facing the internet. For example, the elasticsearch guys decided that it would be a great idea to allow anyone who can access it to run java code on the server at one point...

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#50

This has been covered elsewhere (like on serverfault: http://serverfault.com/a/107346/2557 ) But it comes down to: - Take existing server down immediately. I'm assuming it is not on an isolated network -- so this should really be a priority. - Prep a new patched server (with a smaller attack surface and updated security credentials) - Postmortem the old box on an isolated network. Try to understand how the attacker g…

Also, you thank the reporter profusely for doing the right thing.
Post reply on HN