Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Anonymous person sent proof of SSH access to our production server

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 246 posts

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#51

Post by laumars »

xorbyte wrote: post #48
laumars wrote: post #43

In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH. 1. firewall - only allow SSH connections from trusted static IPs 2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo,…

You assume the breach happened over SSH. This is valuable information to securing SSH, but it's entirely possible the original breach happened over some other service, and there were some other steps involved in the breach before the SSH screenshot was taken.

True but I'm working from the angle that If the breach happened via some other means then they'd need some way to remotely execute code to enable SSH, create valid login credentials, and disable the firewall; in which case they already have a more convenient shell access so gaining access to SSH becomes redundant.

However it's possible that the attacker's screenshot was of a remote shell initiated via some other means and the OP assumed it was via SSH.

Edit: why was this downvoted? If there's an error then I need to be educated. I've spent enough years of my professional life hardening servers to have some idea what I'm talking about, but I'd be an idiot if I didn't listen to the expertise of others. So please correct me rather than downvote me :)

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#52

Post by xorbyte »

caspereeko wrote: post #37

Why would you have and production system exposing SSH to the public? If you must, at least do these steps: - Disable password SSH login - Install root kit scanner, like rkhunter and check if your networked systems are infected. s/he might gained access to other instances in your infra. - Use port scanning on all your instances and check if there is any suspecious rpc port is open that you are not familiar wtih. - Ena…

There's nothing in the OPs post suggesting SSH was exposed to the public, or that the breach happened over SSH. So it's important to secure that, but it's also important to think holistically about the attack surface.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#53

Post by eeZah7Ux »

Do not trust random HN or serverfault answers.

Cleanly shutting down the server can trigger rootkits that might wipe evidence: talk to a professional. Pulling the plug can still remove the ability to observe the behavior of the attacker: talk to a professional. Touching the disks can expose you to the risk of being accused of tampering evidence: talk to a professional.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#54

Post by laumars »

ultramancool wrote: post #49
laumars wrote: post #43

In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH. 1. firewall - only allow SSH connections from trusted static IPs 2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo,…

Keep in mind though that while securing SSH is a good approach, SSH itself is very unlikely to be the route of compromise unless an extremely insecure account were present with weak password auth. It's far more likely that the attacker got legit credentials via another means, web application vulnerability, social engineering or malware attack on company machines, etc. I'd look at the less common applications you run,…

Indeed, however some of my points still secure against that:

1. firewalling to only the sysadmin's IPs,

2. SSH keys + disabling password logins

6. and disabling SSH on internet facing IPs altogether (if possible).

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#55

Post by dimino »

junke wrote: post #45
disease wrote: post #13

Give the person a small reward for pointing out the vulnerability and offer another small reward for suggesting how to fix it.

Maybe hire that person?

Hire someone because they exploited a single vulnerability?

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#56

Post by laumars »

devishard wrote: post #46

To people focusing in on securing SSH: just because the person has SSH access doesn't mean that they got it through SSH. It's possible that they brute forced the password or whatever, but there's a ton of attack surface on a website and many ways they could have gotten access. If they got it through for example an XSS attack and got the SSH password/keys, securing SSH doesn't stop them from doing the same thing again…

I was one of the people offering advice on hardening SSH. I mentioned firewalling sshd to a subset of trusted IPs - which would still secure you against the above attack. I also suggested SSH keys should have passphrases, which would also mitigate against this attack (providing the passphrases are complex enough). And if anyone has root permissions to disable the firewall or change user login credentials, then they don't need to enable SSH (much easier ways to gain interactive shells).

I also love how you can take genuinely helpful posts - after all, it's better to harden SSH regardless of whether this specific attack initially came directly from SSH - and somehow turn those contributions into something negative. God bless internet messageboards.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#58

Post by obi1kenobi »

laumars wrote: post #43

In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH. 1. firewall - only allow SSH connections from trusted static IPs 2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo,…

Consider setting up 2FA for SSH: https://wiki.mozilla.org/Security/Guidelines/OpenSSH#Multi-F...

The rest of that article is very helpful as well!

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#59

Post by terminado »

edelans wrote: post #32
disease wrote: post #13

Give the person a small reward for pointing out the vulnerability and offer another small reward for suggesting how to fix it.

why small ? the reward should be in accordance with the criticity IMHO.

Is "criticity" a word with actual cromulence?

I feel like I've been embiggened by learning it today.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#60

Post by throwaway0727 »

OP here. Thanks for all the responses.

I took action and updated firewall settings (which were too loose), ensured that offsite backups are in place if worse comes to worst, rotated all api keys etc, meanwhile trying to contact the anonymous person. Will rebuild the servers asap as well, super glad that we have properly maintained ansible scripts.

Also will try my best to convince the CEO to allocate some money for professional audit/consultancy since we are no experts in security and to reduce the chances of future incidents.

Trying to do our best and avoid things like SQL injection, XSS, etc but no one is secure after all.

Post reply on HN