Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Anonymous person sent proof of SSH access to our production server

news.ycombinator.com

Post reply on HN
Print view View original

201–210 of 246 posts

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#201

Post by bArray »

>Anonymous person (under a nickname) sent screenshot as a proof that they managed to gain SSH access to our production server. The screenshot is legit, information displayed in it could not be faked without actual access.

Could you upload a redacted version of the email? Get rid of anything that identifies you or the company - the community here might be able to help.

I agree with the majority of the advice here - get the server down immediately after preparing a new, security enhanced version.

Assuming the following:

  * You use a key for access and not a typed password.
  * You haven't done done anything to your distro to compromise security.
There are probably two likely routes of thinking here:

  * The key leak was internal to the company and you need to figure out who it was.
  * The software your company wrote introduced a security vulnerability.
For the first one, you need to devise a method of catching them out - you at least want to know if it was an internal or external source. Assuming this person is smart, they'll probably be using a VPN or TOR so tracking their IP will be useless. Try the following:

  * You could set SSH to only accept incoming connections from your company (and your home IP as backup). Allowing SSH connections from arbitrary places is generally not a good idea.
For the second one, it would be good to know what sever software/libraries you are using as well as versions. `sudo nmap -sA ` your own server so we can see what you have running and possible entry points.

Also:

  * Patch SSH to send you all commands typed - if you've been compromised you at least want to know to what extent.
  * Assume whatever data was on the server is now compromised - databases, passwords, usernames, emails, bank accounts, etc. You need to inform your customer base if their details are leaked. Internal or external they've already copied everything (I would have).
They are just some initial ideas - no doubt somebody will shoot them down but hopefully you can find some use in them.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#202

Post by figgis »

middleca wrote: post #177
justin_vanw wrote: post #109

Enable two factor authentication on your servers, or else two factor on a bastion server and disable external ssh other than from your bastions on all other servers. I strongly recommend YubiKey, it is convenient and cheap and extremely secure. With ssh it is very easy for someone to create an ssh key that does not have a passphrase. With that, it is possible to log into the server with just the key file and nothing…

Came here to say +1 to this, definitely employ a bastion host and make sure that's the only way to SSH to your servers. This can be a little tricky to do correctly if you don't have someone on your team, but it's a valuable way to reduce your surface area to monitor. Installing fail2ban is also a very basic / smart way to discourage brute force SSH attacks on your boxes. Also you could try piping your SSH logs into s…

On fail2ban, I have had more success in being able to stop attacks quickly by using SSHGuard. Quicker easier setup, easier to understand, etc. Is there a significant reason to use fail2ban over sshguard?

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#203

Post by cjcampbell »

cjcampbell wrote: post #199

Came across your question moments after facilitating a security incident dry run for a SaaS with a 6 or 7 person core team (and a lot of sensitive data). Since many of the responses are focusing on the technical and reward/ransom aspect of this, I figured I could offer you some parallel thoughts on how to handle this from the business and liability standpoint. There is quite a bit in this response, so please don't he…

FYI, if you do want to chat. Our new website will include resources, but it hasn't launched yet. Hit me up for now on https://www.linkedin.com/in/clintonjcampbell. Our twitter account, @quirktree is operational though not launched yet. And our website, quirktree.com, should be ready by next Monday with online contact options.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#204

Post by scoot »

emcrazyone wrote: post #158
danesparza wrote: post #3

This has been covered elsewhere (like on serverfault: http://serverfault.com/a/107346/2557 ) But it comes down to: - Take existing server down immediately. I'm assuming it is not on an isolated network -- so this should really be a priority. - Prep a new patched server (with a smaller attack surface and updated security credentials) - Postmortem the old box on an isolated network. Try to understand how the attacker g…

100% agree. Disclaimer I own a data center and have dealt with customer collocated equipment breaches. In addition to the above steps: -> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no -> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root. -> disable password based logins and go to cer…

> Disclaimer I own a data center

I'm curious why you think owning a DC makes you less qualified to respond? Presumably because someone that senior is less in touch with day-to-day security operations?

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#205

Post by rahimnathwani »

scoot wrote: post #204
emcrazyone wrote: post #158

Earlier quoted context omitted.

100% agree. Disclaimer I own a data center and have dealt with customer collocated equipment breaches. In addition to the above steps: -> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no -> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root. -> disable password based logins and go to cer…

> Disclaimer I own a data center I'm curious why you think owning a DC makes you less qualified to respond? Presumably because someone that senior is less in touch with day-to-day security operations?

I think they meant 'disclosure'

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#206

Post by cjcampbell »

ryanlol wrote: post #2

>We are a small company and don't have any security experts, etc. Find one.

This can be a tricky proposition for small companies given the breadth of experience that is needed to adequately address this sort of scenario.

The practical solution for most is to outsource, but it can be challenging to find affordable and high-quality service at a reasonable price.

I'm getting ready to launch in this space. With few exceptions, the services I trust are focused on the glut of opportunities with deep pocketed clients. I can count on my fingers (and maybe toes) the number of colleagues and companies who are willing to pass that gold rush up to deal with the needs of the smaller clients in a meaningful way.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#207

Post by scoot »

rahimnathwani wrote: post #205
scoot wrote: post #204

Earlier quoted context omitted.

> Disclaimer I own a data center I'm curious why you think owning a DC makes you less qualified to respond? Presumably because someone that senior is less in touch with day-to-day security operations?

I think they meant 'disclosure'

Possibly, but that would suggest a conflict of interest which is apparent here. More of an appeal to authority perhaps?

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#208

Post by sredfern2 »

VikingCoder wrote: post #181
emcrazyone wrote: post #158

Earlier quoted context omitted.

100% agree. Disclaimer I own a data center and have dealt with customer collocated equipment breaches. In addition to the above steps: -> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no -> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root. -> disable password based logins and go to cer…

...it's a Windows box. (Just kidding.)

:-)

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#209

Post by voidlogic »

zpallin wrote: post #189
kbenson wrote: post #91

In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of: "Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you woul…

Key here is that you need to figure out how they got in. Then negotiate terms to have them back off. Either way, they're probably being nice about it if they haven't simply 'rm -rf /'d you.

> if they haven't simply 'rm -rf /'d you.

Or dd if=/dev/urandom of=/dev/sda1 bs=8M

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#210

Post by spdustin »

uxp wrote: post #197
spdustin wrote: post #195

Earlier quoted context omitted.

RootKit Hunter [0] works pretty well on most distros to check hashes on files and other potential problems. You can also use a global bash configuration that would log all commands [1] entered into any bash shell to a central log, which could be shipped off-server simultaneously. [0]: http://rkhunter.sourceforge.net [1]: http://askubuntu.com/questions/93566/how-to-log-all-bash-com...

Does RKHunter scan userspace for mtimes? I haven't used it in years, so I'm honestly curious. Back when I did, it was customary to install it side-by-side with Tripwire, which essentially does only that; scan userspace and categorically log changed files based on a configurable severity depending on location (eg, /root/ is high, /var/log/messages is low)

They really serve two different functions. Tripwire and similar tools like aide (I use aide now rather than tripwire, but that's my personal preference, I'm not an infosec domain expert) are file integrity checkers that check files for property changes (including mtime). Tuning out false alarms can result in an admin just turning off the reporting functions of the tools, that's the downside as I understand it.

However, rkhunter has additional logic to specifically seek out rootkits and malware-like behavior, and is more specifically targeted to system file modifications. Combining it with unhide (to compare actual processes running with those visible from userspace) provides a reasonable assurance that nothing nefarious is going on.

They're all part of a spectrum, however. I use scanners like aide alongside rkhunter as well, but I'm the sort of guy that will spend a day tuning the config of aide to avoid constant false alarms.

Post reply on HN