Ask HN: Anonymous person sent proof of SSH access to our production server

It looks like the [flagged] label was incorrectly displayed—users haven't flagged it.

My point was correct to the specific attack you broadly described. Your example required a web server attack that allowed arbitrary code execution and privladge elevation; which is a hugely specific attack and it's pretty fair to say it's unlikely (in the case of gaining root access and then choosing to enable SSH, which I'll get to). Furthermore, and at risk of sounding like a broken record, if an attacker can remotely execute code as root then they have absolutely no need to enable SSH for they already have far easier methods of firing off a remote shell. (To be honest they don't even need root to accomplish this).

This is not a mistake, this is something I've done in practice when auditing security at work. (Preventative pen testing rather than post breach forensics such as this situation calls for. But as I said before, that doesn't mean you cannot take lessons from the former in conjunction with the latter.

Anyhow, we really are just arguing about arguing now, which is an utterly pointless waste of both our time

Contact me if you need a recommendation. I can point you to good security consultants probably within your budget.

Best not to ask why downvoted. Those people's responses will rarely teach you anything. The kind that would will usually reply instead of downvote. Plus, a few already explained to me it's common for a post to get hit with a few negative votes followed by corrective action as other, open-minded people show up. Happens all the time with mine.

Treat is as the worse case scenario, even if you think that guy who got in was a "good guy", some one else might have also visited your box with other intentions.

0) Check your own policies (if you do not have an incident response policy it's a good time to start one), compliance and regulation/legislation specifically around due diligence and notification/disclosure requirements, consult with a legal adviser especially around dealing with who ever informed you about the breach, I would not advise you to deal with the notifying party without doing so, and do not provide them with any compensation until you have spoken to a lawyer.

1) Revoke all credentials which tied to that system, revoke all credentials of all users and services that have accessed that system, revoke all secrets that were stored on that system or the system had access too like certificates and encryption keys.

2) Do not turn off the system unless you can preserver memory, do a full snapshot/image but keep it running keeping the memory intact is important for forensics and a reboot can erase a lot of evidence.

2.1) Isolate the system from your main network with minimal interruption, if you can prevent the NIC on the link from going down do so, if at all possible to mirror the port on your switch do it and enabled pcap for the next 24-48 hours at the least.

2.2) Identify any additional logs from other systems (load balancers, routers, firewall etc.) that could potentially have additional information regarding the breach and preserve them.

3) Do a full integrity analysis and inventory on one of the clone images/snapshots and compare it to your build policy/template and identify any discrepancies.

3.1) Check your current build template/configuration against your own policies (if they do not exist it's a good time to start making them) and best practices and identify any gaps.

3.2) If you received a detailed explanation of how the hack was done check if and what in your policies could or should stop it, if there was nothing implement a new control and add it to your build/config template.

3.3) If you have successfully identified how they got in review any other systems that can be accessible via the same vector.

3.4) Attempt to verify if any other systems were or could be compromised during the breach / vulnerability window and based on your risk assessment make a call to do a full review/rebuild of those system too.

3.5) If this was a common vulnerability get a vulnerability scanning tool (e.g OpenVAS) and scan all your systems.

3.6) If this was more social engineering/i found your SSH creds on GitHub then policies and awareness training should not be taken lightly.

3.7) If this was some sort of super duper NSA grade 0-day (unlikely ;)) notify the maintainer of the SSH software you are using about the breach.

4) Based on the outcome of step 3 either rebuild any machines that were compromised, if the vulnerability cannot be fixed immediately implement mitigating measures (restrict SSH access, implement 2FA, implement a jump box, perform active logging on all connections to the vulnerable machines).

4.1) If you are required or wish to perform a full forensic analysis of the entire incident.

Depending on the outcome of step 0 you may be required to have a qualified 3rd party perform the incident response and a forensic investigation, especially if you do not have in house "certified" people to do it, some EU countries are especially strict about this like Germany. You are also might be required to notify your customers regarding the breach, even if you do not I suggest you do it otherwise you might see "how i hacked XYZ" in a few weeks on HN and realize they are talking about you.