Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SSH: Best practices

blog.0xbadc0de.be

Post reply on HN
Print view View original

71–80 of 123 posts

Re: SSH: Best practices

#71

Post by hackercomplex »

jvehent wrote: post #69
hackercomplex wrote: post #63

Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. In my view there are emerging best practices in this area. There are two ways to reduce this risk and both are controversial: 1. Force developers to only develop software using an SSH ter…

> Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. It's also the least likely to happen. Attackers will have broken deep into your database due to poor webapp security before getting a hold of your ssh private key. Not that SSH security…

I think that unfortunately you're mistaken. If all you do is sit and look at log files all day you can come off with this impression because you see so many scans go by, but that creates a bias in your thinking which doesn't line up with the facts.

A motivated attacker will always tend to try the "low tech" approaches first to rule them out as a first step because it it doesn't require as much sophisticated technical attention and in addition it's a well known fact that human beings are more often than not the weakest link in the chain of security.

It goes something like this:

1. enumerate the employees.. in other words build a list of all their work emails and try to obtain their personal emails and those of their spouses and children.

2. investigate the background of executives so that a compelling phishing email message can be crafted which looks like it originated from high up within the organization.

3. Send targeted phishing emails designed to bait people within the organization to visit webpages that exploit Adobe Flash attack vectors or other browser based vulnerabilities, or perhaps even lure them into installing a trojan directly.

4. after someone in the organization falls victim to the client-side attack, read their emails to learn more about the organization's structure so that your next phising attack can be more refined, specific, and compelling and can possibly utilize a real corporate email address.

5. Rinse and repeat until you eventually gain access to a developer laptop where you can grab production environment SSH keys. Now you own the network without even having to scan a single server and without leaving a trace in the log files.

Re: SSH: Best practices

#72

Post by jvehent »

hackercomplex wrote: post #71
jvehent wrote: post #69

Earlier quoted context omitted.

> Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. It's also the least likely to happen. Attackers will have broken deep into your database due to poor webapp security before getting a hold of your ssh private key. Not that SSH security…

I think that unfortunately you're mistaken. If all you do is sit and look at log files all day you can come off with this impression because you see so many scans go by, but that creates a bias in your thinking which doesn't line up with the facts. A motivated attacker will always tend to try the "low tech" approaches first to rule them out as a first step because it it doesn't require as much sophisticated technical…

You're describing the workflow of an APT attack. Those are extremely costly, take several months to complete and definitely involve highly technical skills. I agree they are a real threat, but not to startups, to Fortune 500 companies.

For startups, the #1 risk is a vulnerability in some web app, or a forgotten admin panel, that leaks the entire database to an attacker. Not a complex attack conducted by a nation-sponsored offensive team.

But again, SSH security matters, you should take it seriously.

Re: SSH: Best practices

#73

Post by hackercomplex »

jvehent wrote: post #72
hackercomplex wrote: post #71

Earlier quoted context omitted.

I think that unfortunately you're mistaken. If all you do is sit and look at log files all day you can come off with this impression because you see so many scans go by, but that creates a bias in your thinking which doesn't line up with the facts. A motivated attacker will always tend to try the "low tech" approaches first to rule them out as a first step because it it doesn't require as much sophisticated technical…

You're describing the workflow of an APT attack. Those are extremely costly, take several months to complete and definitely involve highly technical skills. I agree they are a real threat, but not to startups, to Fortune 500 companies. For startups, the #1 risk is a vulnerability in some web app, or a forgotten admin panel, that leaks the entire database to an attacker. Not a complex attack conducted by a nation-spon…

I think that "APT" is just a fancy new word that describes a very old methodology that has been commonplace since the earliest days of computer crime. If you read about Kevin Mitnick for example he was doing this stuff in his early teens.

I think that you may have a dangerous attitude about it because in modern times it's not a question of whether or not you're a big enterprise or a startup it's a question of whether or not the dataset at the nucleus of your system would be valuable on the black market or not. If an attacker or group of attackers thinks that your dataset could be saleable one day in the future as your company continues to grow then instead of trying to buy your equity on the secondary markets they may invest in trying to "own" your infrastructure now before you become big enough to put your employees thru white-hat training around social engineering.

I suppose my point here is that it does make sense for startups to put their team through proper white-hat training but it doesn't have to be expensive because you can roll your own. What I suspect is that in 10 years or so this kind of anti-social engineering training will be a standard for any IT knowledge workers not just programmers and will likely be part of the job interview process.

We are aruging over something moot though, since we both agree.. take it seriously.

Re: SSH: Best practices

#74

Post by rphlx »

Also, move sshd to a random & rarely-used TCP port. Won't help much against a skilled & determined targeted attack, however the "security through obscurity" is not entirely worthless. Using something other than 22 is quite effective at avoiding spray-n-pray scans and exploits.

For extra credit, set up port knocking.

Re: SSH: Best practices

#75

Post by jvehent »

hackercomplex wrote: post #73
jvehent wrote: post #72

Earlier quoted context omitted.

You're describing the workflow of an APT attack. Those are extremely costly, take several months to complete and definitely involve highly technical skills. I agree they are a real threat, but not to startups, to Fortune 500 companies. For startups, the #1 risk is a vulnerability in some web app, or a forgotten admin panel, that leaks the entire database to an attacker. Not a complex attack conducted by a nation-spon…

I think that "APT" is just a fancy new word that describes a very old methodology that has been commonplace since the earliest days of computer crime. If you read about Kevin Mitnick for example he was doing this stuff in his early teens. I think that you may have a dangerous attitude about it because in modern times it's not a question of whether or not you're a big enterprise or a startup it's a question of whether…

It's all a matter of prioritization. In an ideal world, address all the issues and be perfectly secure. But if you have to choose by priorities, private ssh key compromise is not exactly at the top of my concerns because people are generally careful about their keys.

Re: SSH: Best practices

#76

Post by hackercomplex »

rphlx wrote: post #74

Also, move sshd to a random & rarely-used TCP port. Won't help much against a skilled & determined targeted attack, however the "security through obscurity" is not entirely worthless. Using something other than 22 is quite effective at avoiding spray-n-pray scans and exploits. For extra credit, set up port knocking.

Choosing to only bind SSH to a VPN interface is another option. If you can utilize a VPN that incorporates 2FA then that's even better. Projects such as zerotier are going a long way towards making this kind of thing easier to setup.

Re: SSH: Best practices

#77

Post by yeukhon »

hackercomplex wrote: post #63

Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. In my view there are emerging best practices in this area. There are two ways to reduce this risk and both are controversial: 1. Force developers to only develop software using an SSH ter…

I argue leaking SSH key is not the worst nightmare but definitely one of the top five on anyone's list. I am more worried about source code and database leaks through development cycle. Copied production data to dev environment and clone source code to your laptop. How does Google/Facebook handle this? I can't imagine anyone cloning their 30G or whatever size single-branch repository down to laptop (they may do shallow clone but doesn't matter).

Re: SSH: Best practices

#78

Post by yeukhon »

sitharus wrote: post #28
carlisle_ wrote: post #23

>Do not SSH cross-server This doesn't make sense. You can safely setup SSH agent forwarding to ssh from server to server without storing your ssh private key anywhere but your local host.

If the server you're forwarding your agent to is compromised it can now talk to your agent. This point could also be 'assume gateway servers are compromised'.

But the article suggests to use ssh-agent.

Re: SSH: Best practices

#79

Post by shawn-furyan »

jvehent wrote: post #70
EvanPlaice wrote: post #68

Earlier quoted context omitted.

3. Never grand direct SSH to access the the staging/production environment. Automate the build and deployment environment on a system that developers don't have direct access. Instead of pushing changes, the bot pulls a specified release branch, builds, tests, and deploys the code. All without human interaction. If malicious code were somehow introduced from a developer's environment, it would be recorded and reflect…

> 3. Never grand direct SSH to access the the staging/production environment. Most complex systems fail in unexpected ways, and the best way to debug is to grant the devs access to production machines. It can be temporary, monitored and through a secure channel, but it's still something most organizations can hardly live without.

I think that this issue is potentially moot if you've transitioned to immutable deploys. Under such a system, arbitrage steps would tend to change from the standard have an SA/Dev ssh in and muck around until the issue is found. Instead, first step may be just redeploying (in case of transient/intermittent issues that are disrupting service), and then checking out the production system locally to do root cause analysis.

So yeah, most organizations couldn't do this today, because they don't have the deploy process in place. But when an organization has made the switch, then it is much more realistic to assume that it can get along without granting ad hoc ssh access to production systems.

Re: SSH: Best practices

#80

Post by superuser2 »

hackercomplex wrote: post #63

Developer laptop compromise is probably the biggest security risk that any startup company faces, because the developer laptop is an uncontrolled environment with a lot of "attack surface" which may have been previously compromised. In my view there are emerging best practices in this area. There are two ways to reduce this risk and both are controversial: 1. Force developers to only develop software using an SSH ter…

(Disclaimer, unicorn employee.) I'm confused. Are you implying people write company code on personally owned laptops? That's insane!

1. Company walks you through setting up FileVault (with key escrow) and VPN client, generating and uploading SSH public key immediately after unboxing laptop.

2. OneLogin + Duo (or pick your SSO/2FA scheme) for everything - internal webapps, GMail, etc.

3. SSH keys managed by Puppet.

4. 2FA verification of SSH logins (using pam-interactive), through a bastion (provided .ssh file makes it transparent), with OSX's default SSH agent, with pretty much all of those best practices configured except smartcards/certs.

5. Engineers have SSH access only to utility/development boxes. You can deploy code to production through a webapp (identifying the commit ID) but you can't get a shell as your application's user, and certainly not root.

6. Webapps moving behind a VPN.

In the rare case that you need to debug in production beyond what you can get from metrics/logs, you pair with a "blessed"/senior sysadmin type.

Post reply on HN