Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Anonymous person sent proof of SSH access to our production server

news.ycombinator.com

Post reply on HN
Print view View original

181–190 of 246 posts

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#181

Post by VikingCoder »

emcrazyone wrote: post #158
danesparza wrote: post #3

This has been covered elsewhere (like on serverfault: http://serverfault.com/a/107346/2557 ) But it comes down to: - Take existing server down immediately. I'm assuming it is not on an isolated network -- so this should really be a priority. - Prep a new patched server (with a smaller attack surface and updated security credentials) - Postmortem the old box on an isolated network. Try to understand how the attacker g…

100% agree. Disclaimer I own a data center and have dealt with customer collocated equipment breaches. In addition to the above steps: -> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no -> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root. -> disable password based logins and go to cer…

...it's a Windows box.

(Just kidding.)

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#182

Post by dozzie »

dimino wrote: post #180
dozzie wrote: post #172

Earlier quoted context omitted.

Yes, the one. It was (is? don't know, I don't use Apple products) a flaw in how Homebrew and its homepage operates.

Didn't they fix it?

As I remember, they argued that this is perfectly fine and there's totally no problem with that.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#183

Post by more_corn »

obi1kenobi wrote: post #58
laumars wrote: post #43

In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH. 1. firewall - only allow SSH connections from trusted static IPs 2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo,…

Consider setting up 2FA for SSH: https://wiki.mozilla.org/Security/Guidelines/OpenSSH#Multi-F... The rest of that article is very helpful as well!

We set up yubikey. Cheap, fast, easy.

Foxpass also offers a nice audited, limited duration 2fa ssh auth service.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#184

Post by eonw »

emcrazyone wrote: post #158
danesparza wrote: post #3

This has been covered elsewhere (like on serverfault: http://serverfault.com/a/107346/2557 ) But it comes down to: - Take existing server down immediately. I'm assuming it is not on an isolated network -- so this should really be a priority. - Prep a new patched server (with a smaller attack surface and updated security credentials) - Postmortem the old box on an isolated network. Try to understand how the attacker g…

100% agree. Disclaimer I own a data center and have dealt with customer collocated equipment breaches. In addition to the above steps: -> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no -> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root. -> disable password based logins and go to cer…

just a note: intelligent exploiters hide their files inside of yours, so the -mtime is useless in many cases, they will set the mtime of their upload to match the rest of the folder they hide in.

command history is also easy to alter if you know what you are doing.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#185

Post by more_corn »

MichaelGG wrote: post #169
laumars wrote: post #81

Earlier quoted context omitted.

I'm not a fan of that option because it's causes an inconvenience for legitimate usage without offering any real security benefits. If you've hardened sshd then the biggest benefit you get from changing the port number is reducing the size of your log files. If you've not hardened sshd then you're just one nmap (et al) scan away from being in the same boat as you were previously. If you really want to hide SSH from s…

How is port knocking not, I dunno, at least 10x more inconvenient for users? FWIW, I've never seen a brute force attack on SSH other than on port 22. Most likely, a targeted attacker would realise if you change the port, you're probably not going to have a trivial password.

I agree, I set up port knocking and it's a huge pain in the butt (so much so that I never moved it out of testing). It also lacks cryptographic verification. It's a lot of trouble for a not quite ideal solution.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#186

Post by laumars »

MichaelGG wrote: post #169
laumars wrote: post #81

Earlier quoted context omitted.

I'm not a fan of that option because it's causes an inconvenience for legitimate usage without offering any real security benefits. If you've hardened sshd then the biggest benefit you get from changing the port number is reducing the size of your log files. If you've not hardened sshd then you're just one nmap (et al) scan away from being in the same boat as you were previously. If you really want to hide SSH from s…

How is port knocking not, I dunno, at least 10x more inconvenient for users? FWIW, I've never seen a brute force attack on SSH other than on port 22. Most likely, a targeted attacker would realise if you change the port, you're probably not going to have a trivial password.

Port knocking is more inconvenient but its also more secure. I'm not advocating it though, just example how low down the scale I consider non-standard ports as a sensible recommendation.

Random IPs are unlikely to be port scanned but any reasonably popular site will.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#187

Post by acd »

I have had to clean up similar in previous job helping tracking down a larger intrusion. Call the police and ask for advice

Here is some server advice, given that the server does not have ransomware.

  1) Save the memory of the machine if possible /dev/mem to disk it may have proof how the hacker got in.
  2) Save process list of the machine
  3) Save the netstat -plunt TCP/UDP output to file, you want to see which TCP/UDP connection the machine have before its turned off. 
  4) Check for any strange process in ps list
  5) If you see any strange process visit /proc/PID and check CWD current working directory and start commands.
  6) List kernel modules with lsmod and dump to file
  7) Power off the machine with the power button, reason you do not want to run any of the normal powerdown scripts
  8) Take a system image of the disk to an external hard disk for proof. This is important that you do not tamper with file access/last modification,change time.
  9) Pull out the network cable
  10) Power on the server again, backup all data files to external media
  11) Read log files
  12) Wipe the hard disk and reinstall the operating system with latest version
  13) Change root and account passwords
  14) Use an SSH key
  15) Lock down ssh port 22 to known IP numbers
  16) Apply all security patches, operating system and applications, make sure applications are running latest patched release
  17) Restore data backup
  18) Deploy host intrusion detection system, HIDS
  19) Send logs to an external machine
Optional steps

  20) If authoritative suggest it try contact the person and ask how he/she got in.
  21) Make a copy of the system image from step 8
  22) Examine system image and logs with something like Autospy from Sleuthkit, you can check last access times, read web and ssh logs.
  23) Ask upstream ISP providers if they have ssh connection logs. Ask if you can get them.
Additional resources Mozilla how to secure SSH https://wiki.mozilla.org/Security/Guidelines/OpenSSH Debian 5.1 how to secure ssh https://www.debian.org/doc/manuals/securing-debian-howto/ch-... Sans top twenty security controls https://www.sans.org/media/critical-security-controls/critic...

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#188

Post by Symbiote »

giis wrote: post #129

My suggestions: If your server has confidential/money related info. take down the server. If not , wait because even if you bring up new server. How do you know , he won't crack it again? - Check your /var/log/messages & audit.log & ssh/d.log - Check lastb & last command outputs - Take dump of network connections. (netstat) - find out his 'tty' and spy on him! with something like sysdig or using strace ( http://serve…

> - Check lastb & last command outputs

I wasn't aware of lastb, thanks.

I'm impressed that my home server has logged over 650,000 failed logins since 1 July, and a couple of machines I administer in a university over 300,000 each. That's every three-four seconds for the home server.

That's quite a lot of bandwidth, worldwide.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#189

Post by zpallin »

kbenson wrote: post #91

In addition to the wonderful technical advice already here for how to deal with the server, there is the question for how to deal with the anonymous person. If the proof contained the method of exploit I suggest something along the lines of: "Thank you for bringing this problem to our attention! We are taking steps to resolve the problem now, but would like to reward you for your work. If you let us know how you woul…

Key here is that you need to figure out how they got in. Then negotiate terms to have them back off. Either way, they're probably being nice about it if they haven't simply 'rm -rf /'d you.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#190

Post by zpallin »

danesparza wrote: post #3

This has been covered elsewhere (like on serverfault: http://serverfault.com/a/107346/2557 ) But it comes down to: - Take existing server down immediately. I'm assuming it is not on an isolated network -- so this should really be a priority. - Prep a new patched server (with a smaller attack surface and updated security credentials) - Postmortem the old box on an isolated network. Try to understand how the attacker g…

You still don't know how they got in, and since they're not doing anything nefarious (that OP knows of) it's probably easier to pay them off to inform OP how they got in.
Post reply on HN