Live data from Hacker News

Ask HN: Anonymous person sent proof of SSH access to our production server

news.ycombinator.com

191–200 of 246 posts

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#191
post #36

Step 1: Try to get in contact with the person and see if he/she is willing to help you share details on entering your systems. Thank this person and see if you can provide a reward. Step 2: Next step is setting up new systems, and start from scratch. Install the systems, start with basic system hardening and up-to-date software packages. Use https://github.com/CISOfy/lynis to validate your configuration. Do not have…

+1 for link to lynis. I had never heard of that before!

You are replying to the author of lynis, might as well thank him for writing it :)

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#192
post #184

Earlier quoted context omitted.

100% agree. Disclaimer I own a data center and have dealt with customer collocated equipment breaches. In addition to the above steps: -> disable root being able to login inside your sshd_config file. Make sure PermitRootLogin no -> rename the root account too so if they are using an exploit based on user authentication then perhaps they won't be able to elevate to root. -> disable password based logins and go to cer…

just a note: intelligent exploiters hide their files inside of yours, so the -mtime is useless in many cases, they will set the mtime of their upload to match the rest of the folder they hide in. command history is also easy to alter if you know what you are doing.

So the only way to detect exploits is to scan the server regularly and log the mtime and the file size, and look for changed files that shouldn't have changed?

Re the command history, is there any way around them wiping it - e.g. piping all .bash_history entries to an append-only store?

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#193
post #98

Earlier quoted context omitted.

God forbid. No hire if they cannot reverse binary tree on the whiteboard using angular react e6.

The guy that couldn't "reverse a binary tree" came from a team that turned out to be incompetent (in operations field; they didn't see anything wrong with distributing all the packages with software using HTTP without SSL/TLS nor any cryptographic signature).

To be fair, some major Linux distros also distribute using http e.g. go to https://www.debian.org/ and check the Download link at the top of the page: it is http.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#194
post #43

In terms of hardening against SSH attacks, the principles are quite simple. Your business case might mean that some of the following cannot be applied, but there's plenty of measures below that you can use to harden SSH. 1. firewall - only allow SSH connections from trusted static IPs 2. Use SSH keys then disable password logins. Lots of guides online to create keys, so I'll just cover the 2nd point: as root or sudo,…

Keep in mind though that while securing SSH is a good approach, SSH itself is very unlikely to be the route of compromise unless an extremely insecure account were present with weak password auth. It's far more likely that the attacker got legit credentials via another means, web application vulnerability, social engineering or malware attack on company machines, etc. I'd look at the less common applications you run,…

I was wondering.....

If someone in the datacenter can image the VM and mount it some place in their own machine, reset the ssh rsa key, etc.

Is it good enough to "produce" the proof of the hack?

If so, than no amount of "clean up" can fix the issue, right?

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#195
post #192
post #184

Earlier quoted context omitted.

just a note: intelligent exploiters hide their files inside of yours, so the -mtime is useless in many cases, they will set the mtime of their upload to match the rest of the folder they hide in. command history is also easy to alter if you know what you are doing.

So the only way to detect exploits is to scan the server regularly and log the mtime and the file size, and look for changed files that shouldn't have changed? Re the command history, is there any way around them wiping it - e.g. piping all .bash_history entries to an append-only store?

RootKit Hunter [0] works pretty well on most distros to check hashes on files and other potential problems.

You can also use a global bash configuration that would log all commands [1] entered into any bash shell to a central log, which could be shipped off-server simultaneously.

[0]: http://rkhunter.sourceforge.net

[1]: http://askubuntu.com/questions/93566/how-to-log-all-bash-com...

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#196
post #82

Earlier quoted context omitted.

Alternatively, ask them for their preferred payment method and don't force your FOTMcoin on them.

I assume he suggested Bitcoin because it's still a good way to handle money anonymously. If the person that breached the server wants to remain anonymous but get paid, that's pretty much the only way to go.

Except for the fact that the transaction becomes a permanently public record.

You could also get looped into tax evasion if the person you paid doesn't report the bug bounty income. You can still maintain anonymity by not announcing the person who found it, but it's not a good idea to give money to anonymous people for work.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#197
post #192

Earlier quoted context omitted.

So the only way to detect exploits is to scan the server regularly and log the mtime and the file size, and look for changed files that shouldn't have changed? Re the command history, is there any way around them wiping it - e.g. piping all .bash_history entries to an append-only store?

RootKit Hunter [0] works pretty well on most distros to check hashes on files and other potential problems. You can also use a global bash configuration that would log all commands [1] entered into any bash shell to a central log, which could be shipped off-server simultaneously. [0]: http://rkhunter.sourceforge.net [1]: http://askubuntu.com/questions/93566/how-to-log-all-bash-com...

Does RKHunter scan userspace for mtimes? I haven't used it in years, so I'm honestly curious. Back when I did, it was customary to install it side-by-side with Tripwire, which essentially does only that; scan userspace and categorically log changed files based on a configurable severity depending on location (eg, /root/ is high, /var/log/messages is low)

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#198

OP here. Thanks for all the responses. I took action and updated firewall settings (which were too loose), ensured that offsite backups are in place if worse comes to worst, rotated all api keys etc, meanwhile trying to contact the anonymous person. Will rebuild the servers asap as well, super glad that we have properly maintained ansible scripts. Also will try my best to convince the CEO to allocate some money for p…

From your post here, you clearly have only rudimentary security knowledge at absolute best.

You need to bring an expert and/or firm on-site.

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#199
Came across your question moments after facilitating a security incident dry run for a SaaS with a 6 or 7 person core team (and a lot of sensitive data). Since many of the responses are focusing on the technical and reward/ransom aspect of this, I figured I could offer you some parallel thoughts on how to handle this from the business and liability standpoint.

There is quite a bit in this response, so please don't hesitate to ask if you want to talk this through directly. I can stand to offer 30min to provide some connections and make sure you feel equipped to go it with your current team. No cost of course and no nagging to get you on as a paying customer!

1. Insurance - Assuming you have a policy geared toward technology companies, you may have coverage for "cyber" incidents. Look at your policy or call your broker to find out how to initiate coverage.

a) You probably need to tell them sooner rather than later that you are investigating an incident. b) Ask for a referrals to the following specialities: tech/privacy law, forensics or security operations, and breach resolution vendors. c) Find out whether the policy requires you to obtain a referral to use paneled providers (most do from my experience). d) Find out any other requirements that might determine your ability for coverage down the line.

2. Legal (assuming American or similar legal system) - Given what I know about your situation, I'd suggest that you retain a lawyer experience with security and breach response. This doesn't have to break the bank. The key point is to establish privilege and get help from someone who can navigate the statutory/liability landscape (it's a shit show). I can recommend one or two that are sensitive to the needs of small business clients.

3. Forensics - If you have insurance coverage that will apply to this scenario, there is little reason not to contact a forensic or security specialist to validate your cleanup and ensure that the infrastructure is totally buttoned down going forward. Definitely follow the rest of this thread and digest the recommendations you're getting.

4. Customers/ Stakeholders - Your customers and stakeholders will appreciate clear communication, whether during the investigation/response or after the matter. The lawyer I mentioned above will be able to help you think about communication with these parties as well as timing. If the incident turns into something more than it is right now, you'll be thankful for thinking this dimension through carefully.

5. Intruder - Don't react carelessly. You don't know their motives or further plans. Is this a gray hat who is just wanting you to shore up your system. I'd be inclined to work with them and possibly even offer a reward. But I wouldn't go there without consulting legal counsel. Is this more about taunting or warming you up for a ransom request? Then get some folks with experience in your corner from the start. And be aware, your insurance might even cover a ransom request.

6. Law Enforcement - Most of us prefer not to go here, but keep it as an option. Talk to your attorney about it.

7. Documentation - Start keeping detailed notes. What are the steps you're taking to assess the situation? Who are you consulting? How are you measuring the risk? How are you preventing further damage? A breach coach can help you ask the right questions and record the information most needed to protect you and your customers.

8. Notification - If you're in the US or EU, you very likely have breach notification requirements. Unless you are certain the intruder did not take or view any protected data (pretty much anything personal or payment related), I would treat the incident as a breach. In the US, you have 47 different state laws dictating notification requirements plus a few federal. Breach resolution services to the rescue if you have insurance. Breach coaches and lawyers may handle the little cases themselves.

9. Follow up - a) Summarize the process in a short report and keep it in case you discover related damage down the line (or get sued). b) Close the incident out with insurance and provide any documentation they need to process the claim. c) Put a basic, written, practical plan together. d) Build a relationship with a knowledgeable insurance broker (I can recommend a few). e) Review, or update your insurance policy (or buy one). f) If you can afford it, keep a tech attorney on retainer. g) Set up a bug bounty through an established program (and if this intruder is well-intentioned, encourage them to use it in the future)

Re: Ask HN: Anonymous person sent proof of SSH access to our production server

#200
post #8

[Edit: not-OP] As the 2 comments so-far have suggested getting security experts, where would be a good place to source security experts? I'm envisioning 2 kinds: * Consultant, working for a fee (with retainer?); * Independent, may be consultant, but could also be someone currently looking for a new permanent role and would bring welcome diversity/expertise to a small team - potentially illiquid / poorly matched hirin…

Addressed some of this in my main response. A lot of companies in this space will have cyber coverage that includes: breach coaches, privacy/security lawyers, forensic specialists, security operations consultants, and companies to handle mandatory breach notifications.

I tend to work mainly with small businesses and professionals in healthcare, law, etc to avoid and prepare for incident response, but I also work with clients in the middle of a breach.

In lieu of someone like me or a dedicated breach coach/resolution company, there are a handful of good attorneys who understand tech/startup culture and even enough of the technical intricacies to provide a good response.

Our new website will include resources, but it hasn't launched yet. I'm often available to do a short call at no cost to discuss level of effort and get a company or startup moving down the right path. If you want to take me up on that, hit me up on https://www.linkedin.com/in/clintonjcampbell now (or quirktree.com starting next Monday).

Post reply on HN