Ask HN: Anonymous person sent proof of SSH access to our production server • Hacker News Board

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#151

Post by nickpsecurity » Wed, Jul 27, 2016, 6:07 PM UTC

CiPHPerCoder wrote: post #74

web007 wrote: post #47

Earlier quoted context omitted.

It's the same concept as auditors, there's the big 4 that you've probably heard of, and a ton of other, smaller firms with varying quality. You could go with a known firm like iSec Partners, Matasano (now NCC) or Mitnick Security. They won't be cheap - at worst they may be able to refer you to some other reputable firm if your budget is limited.

> or Mitnick Security Please no. Not Mitnick. I'd rather funnel clients towards my competitors than Kevin Mitnick. He's a skilled social engineer, and his greatest social engineering success was manipulating the media into believing he speaks for hackers in general. He is not a programmer, his opinions on cryptography aren't insighful, etc. His only skill is deception.

A nice example is when I used to point out OpenVMS security benefits. Better architecture, higher quality focus, and few things running by default led to fewer vulnerabilities and more containment than competing systems. At least one person always had a recurring counterpoint: "but didnt Mitnick hack all kinds of VMS systems?"

If by hack, you mean use conned or password-guessed credentials to get in... Not my definition of the word.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#152

Post by andrewaylett » Wed, Jul 27, 2016, 6:07 PM UTC

purerandomness wrote: post #139

_kst_ wrote: post #115

Earlier quoted context omitted.

If you're going to offer a reward, I suggest making it conditional on revealing how they got in. (This also assumes you're able to contact the person who sent the e-mail.)

> making it conditional on revealing how they got in Do you mean obligatory ? If you really mean conditional, could you elaborate why?

In this case, the payment is conditional on revealing the method: that is, unless the method is revealed (condition met) the payment will not be made.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#153

Post by niij » Wed, Jul 27, 2016, 6:11 PM UTC

throwaway0727 wrote: post #126

kbenson wrote: post #91

In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of: "Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you woul…

Unfortunately, it's the later - no details of exploit, just a proof. If this comes to ransom, rather than unethical/unexperienced gray hat thing, are there any good steps to take? Or hiring an expert consultancy is probably the only good option here?

I would recommend taking your oldest backup offline and storing it indefinitely, in case later backups are corrupted. Make sure you turn on verbose firewall logging as well.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#154

Post by devishard » Wed, Jul 27, 2016, 6:16 PM UTC

laumars wrote: post #134

devishard wrote: post #117

Earlier quoted context omitted.

Okay sure. They can figure out how the attacker got access and fix that, AND harden SSH. They should also audit their logs, install updates, use a linter on their JavaScript, use version control, use a library to sanitize SQL inputs, force HTTPS, do code reviews, and 100 other good development practices that have nothing to do with the problem at hand. I'm glad we agree on that. Given that the OP doesn't know how to…

My point was correct to the specific attack you broadly described. Your example required a web server attack that allowed arbitrary code execution and privladge elevation; which is a hugely specific attack and it's pretty fair to say it's unlikely (in the case of gaining root access and then choosing to enable SSH, which I'll get to). Furthermore, and at risk of sounding like a broken record, if an attacker can remot…

> My point was correct to the specific attack you broadly described.

But you didn't correct it, you proposed a solution that didn't address the attack I described.

> Your example required a web server attack that allowed arbitrary code execution and privladge elevation; which is a hugely specific attack and it's pretty fair to say it's unlikely (in the case of gaining root access and then choosing to enable SSH, which I'll get to).

The XSS attack is just an example of a vulnerability that wouldn't be addressed by hardening SSH. There are plenty of other vulnerabilities that wouldn't be addressed by hardening SSH.

> Furthermore, and at risk of sounding like a broken record, if an attacker can remotely execute code as root then they have absolutely no need to enable SSH for they already have far easier methods of firing off a remote shell. (To be honest they don't even need root to accomplish this).

There's a good reason to demonstrate SSH access even if they have root access: they might want to show their capabilities without exposing how they gained those capabilities (because knowing how they gained those capabilities would allow the OP to fix the problem).

> This is not a mistake, this is something I've done in practice when auditing security at work.

Just because you've made mistakes in practice when auditing security at work doesn't mean they aren't mistakes.

This isn't even time for a security audit. OP first really needs to do some forensics. A security audit should happen, but it can wait until the vulnerability has been found and fixed.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#155

Post by danesparza » Wed, Jul 27, 2016, 6:17 PM UTC

mboelen wrote: post #36

Step 1: Try to get in contact with the person and see if he/she is willing to help you share details on entering your systems. Thank this person and see if you can provide a reward. Step 2: Next step is setting up new systems, and start from scratch. Install the systems, start with basic system hardening and up-to-date software packages. Use https://github.com/CISOfy/lynis to validate your configuration. Do not have…

+1 for link to lynis. I had never heard of that before!

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#156

Post by netik » Wed, Jul 27, 2016, 6:26 PM UTC

JasuM wrote: post #83

calgoo wrote: post #75

Earlier quoted context omitted.

Also, to get rid of 60% of the script kiddies, change the port to something high 1000+ (or even better 10000+). Its not a great option, but it does lower the amount of automated attacks that you get.

I'd be very cautious before moving SSH to a non-privileged port (over 1024). Any user on the server might start their own SSH server on the port assuming the real SSH server is dead. While this is hard to exploit (needs access to normal user, needs to kill real SSH server, need to get around SSH server key checking), it still is at least a theoretical reduction in security.

A user can always start their own SSH server. Just because you've decided to move it to a different port doesn't really encourage them. I suppose you could make this a bit more difficult for them by removing compilers (no really, you don't need compilers everywhere) and making sshd owned by root, mode 700...

However, proper ingress filtering or local iptables/pf rules would stop any unwanted inbound traffic from reaching your server, and you should definitely be using ingress and egress filtering on your network.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#157

Post by ReFruity » Wed, Jul 27, 2016, 6:37 PM UTC

I wonder if if that person just emailed them vulnerability, then it would just get patched within a day and he got only $1000.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#158

Post by emcrazyone » Wed, Jul 27, 2016, 6:38 PM UTC

danesparza wrote: post #3

This has been covered elsewhere (like on serverfault: http://serverfault.com/a/107346/2557 ) But it comes down to: - Take existing server down immediately. I'm assuming it is not on an isolated network -- so this should really be a priority. - Prep a new patched server (with a smaller attack surface and updated security credentials) - Postmortem the old box on an isolated network. Try to understand how the attacker g…

100% agree. Disclaimer I own a data center and have dealt with customer collocated equipment breaches. In addition to the above steps:

-> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no

-> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root.

-> disable password based logins and go to cert based auth. This will shutdown brute force attacks.

-> lower MaxAuthTries in sshd_config to something like 1 or 2 to help slow down attackers.

-> change the port you listen on. A little security through obscurity while not very effective might slow future casual port scanners that are testing a single port. In practice I've seen this really eliminate a lot of reconnaissance or farming type activities.

-> Make sure you're openssl and openssh are at the latest stable releases.

-> If the perp is coming from the same IP, you can use an iptables rule to block the netblock. Again, not perfect but may help slow things down.

-> grab shell history of all the user accounts on the box. Example ~/.bash_history. This is more reconnaissance but may be helpful if they are sloppy - you might see what they were doing on the box.

-> look for any modified files or new ones. Obviously logs will show up in the list but you're looking for things that should not be there. Example: find / * -mtime -60 -print where 60 is how many days ago you want to go back.

-> look at chron file to see if any timed delay bombs exist.

-> look at ps -aux output for any running processes that don't make sense

-> look at iptraf for any suspicious traffic to IPs you can't reconcile.

Good luck!

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#159

Post by kogus » Wed, Jul 27, 2016, 6:38 PM UTC

kbenson wrote: post #147

throwaway0727 wrote: post #126

Earlier quoted context omitted.

Unfortunately, it's the later - no details of exploit, just a proof. If this comes to ransom, rather than unethical/unexperienced gray hat thing, are there any good steps to take? Or hiring an expert consultancy is probably the only good option here?

I can't comment on the correct approach in that case, I'm under qualified. I would urge you to make sure you have good backups in a location that can't be compromised (as in, you won't wake up tomorrow to fine them all deleted). If your system already supports this, all the better. Keep in mind the worst case scenario here is that every production server is wiped, which is essentially close to the situation of a natu…

Total data loss isn't the worst case scenario, in my opinion. Quietly interacting with your site, contacting your customers to abuse their trust in you, etc.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#160

Post by laumars » Wed, Jul 27, 2016, 6:39 PM UTC

devishard wrote: post #154

laumars wrote: post #134

Earlier quoted context omitted.

My point was correct to the specific attack you broadly described. Your example required a web server attack that allowed arbitrary code execution and privladge elevation; which is a hugely specific attack and it's pretty fair to say it's unlikely (in the case of gaining root access and then choosing to enable SSH, which I'll get to). Furthermore, and at risk of sounding like a broken record, if an attacker can remot…

> My point was correct to the specific attack you broadly described. But you didn't correct it, you proposed a solution that didn't address the attack I described. > Your example required a web server attack that allowed arbitrary code execution and privladge elevation; which is a hugely specific attack and it's pretty fair to say it's unlikely (in the case of gaining root access and then choosing to enable SSH, whic…

> The XSS attack is just an example of a vulnerability that wouldn't be addressed by hardening SSH. There are plenty of other vulnerabilities that wouldn't be addressed by hardening SSH.

Well durr! We could be here all night listing things that SSH hardening wouldn't secure against. I never suggested it was a silver bullet to fix all security needs (which seems to be the faux argument you're accusing me off).

I'm loving all the personal attacks too. You cannot compete on an intellectual level so you make baseless accusations about my professional capabilities instead. God bless Internet message boards....