Live data from Hacker News

How I spend my first 5 minutes on a server

plusbryan.com

231–240 of 355 posts

Re: How I spend my first 5 minutes on a server

#231
post #221

Earlier quoted context omitted.

If someone logs in directly as root, how do you track root access? Given that you should be discouraging root use and using sudo or scripts to automate admin tasks, why are you encouraging admins to log on as root?

If someone logs in directly as root, how do you track root access? By their SSH-key and IP address (both of which are logged to a remote syslog-server). Given that you should be discouraging root use and using sudo or scripts to automate admin tasks, why are you encouraging admins to log on as root? That is a false premise. I'd never encourage anyone to use "sudo" because it's a source of errors; people get the escap…

Logging the ssh key used sounds interesting - not something I've tried before.

I think you're doing something wrong with sudo. I'm not talking about "reentering your password" in the Ubuntu or Mac sense. I'm talking about giving access to specific limited commands.

And what distro are you running where local privilege escalation is common?

Re: How I spend my first 5 minutes on a server

#232

The premise of this thing is not good advice. 1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one. 2) Everything else listed in this document should be configured by a configuration management system. 3) "User account sync tools" should have no place in a modern infrastructure, you should use your configurat…

I'm a good developer, but setting up systems is something I do so infrequently that I always have to relearn it, and I am positive I could be doing things better. Do you have a link to an article that explains how to go about performing some of these ideas you mention?

Re: How I spend my first 5 minutes on a server

#233

The premise of this thing is not good advice. 1) Your first couple minutes on a server should be used to install a configuration management client, if your bootstrap policies somehow don't already install one. 2) Everything else listed in this document should be configured by a configuration management system. 3) "User account sync tools" should have no place in a modern infrastructure, you should use your configurat…

You are right, this is a better path But sometimes, and especially for servers that will be delivered to the customer afterwards, it's not practical to use a configuration management tool. Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong. Also, Chef/Puppet type solutions may be overkill for some tasks, fabric takes care of the easier cases for example.

>Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong.

You're right -- many used cfengine. Still others used a custom 'config' rpm / deb that deployed all of these files everywhere.

Automated configuration makes sense not just for repeatability, but for auditability and documentation. Especially when you are going to 'hand the server over', the next admin should be able to know what you've changed.

Also, disallow password-based access to everything (use the keys, Luke.)

Re: How I spend my first 5 minutes on a server

#234
post #134

Earlier quoted context omitted.

> Logs wears on the hard drive, so one might want to install fail2ban to lower the number of writes to it. Or change the ssh port.

Another "security through obscurity" pointless trick. If I'm attacking your server and it's not running SSH on port 22, it's a very short matter of time until I've found it anyway.

I don't understand this objection.

If you've got a physical safe that you keep money or valuables in, do you put it out at the kerbside so people can have a go at cracking it, or do you hide it inside in an obscured location to make it more difficult to have a go?

Surely it's the same with services on your servers, why needlessly advertise their presence. It's a tiny veneer of security but it is still more secure surely.

Re: How I spend my first 5 minutes on a server

#235
post #129

Earlier quoted context omitted.

Why do you say a VPN server is more secure? Which one? I, for one, trust ssh more than any other software wrt security, especially with password login disabled. Disclaimer: I am not a security expert.

I, also, trust SSH more than any other software. But it is still worth adding an additional layer of security in front of SSH to help protect from exploits. Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack. Your servers allow SSH from anywhere on the internet, and are ev…

Your servers allow SSH from anywhere on the internet, and are eventually discovered and exploited. Mine, which will only allow SSH connections from my VPN bastion host, are effectively invisible to the attacker and will not get exploited

So, just to see if I'm reading you right: you're using a VPN in the place of an SSH jump box, not making a judgement about the fitness or trust placed in your VPNd over your SSHd.

Re: How I spend my first 5 minutes on a server

#236

Earlier quoted context omitted.

I was in the same position - too many distinct environments for bash/Fabric, too little time to learn Chef/Puppet/CFEngine. Ansible [1] seems like a good compromise: you get the simplicity (runs over SSH) and host targeting of Fabric with the declarative nature and idempotency of the more complex tools. You can start with all-in-one "playbooks" [2], then split out tasks, handlers, Jinja2 templates, files, and variabl…

I'm using fabric quite frequently, and am trying to understand what makes a configuration management tools a much better choice. I'm currently using fabric for anything from bootstrap a new environment from scratch, via restoring a snapshot from backups, to pushing code updates stored on git. Perhaps I'm being really daft, but it always evades me why something as simple as sudo("apt-get install -y ") needs to be repl…

I'm not that familiar with Fabric, so maybe it can do some of the things I mention here out of the box without a lot of custom code. I'm only familiar with Capistrano and I associate them in my head, perhaps naively.

Your example is great to illustrate why configuration management is so powerful.

Most importantly, your example will upgrade the package to the latest version every time you run it (which is probably not what you intended, but maybe it is). I'm sure there is an idempotent command for each of apt, yum, ports, homebrew, etc, but I have better things to do with my time than figure that all out.

In puppet your command would be:

    package { 'foo': ensure => installed }  # or 'latest'
What if you also have RedHat servers? Now you need to figure out the right yum command to use to perform the equivalent. Or your developers Macs? Homebrew. Chef/Puppet are aware of the environment they run in and install appropriate packages using the appropriate package management script with minimal alteration to the underlying recipe.

For RedHat systems, it would be:

    package {'foo' : ensure => installed }
On a Mac, it would be:

    package {'foo' : ensure => installed }
On that Arch Linux machine that your devs are running, it would be:

    package {'foo' : ensure => installed }
Now, let's throw some dynamism into the mix...

What if you want to build an haproxy configuration file that pulls the list of hosts from an authoritative source? Do you "just" build it up from scratch every time and replace the file on every run of your script? With chef/puppet, you can pull that information in from your node classifier using the same query for dev, staging, and production and have the same recipe work for multiple environments. In some environments, the haproxy config has now changed, so I want to tell haproxy to reload. But in other environments, the config stayed the same, so I don't want it to reload. The file and service don't get touched if they didn't change.

What if a service has multiple files associated with it that when one or multiple of them change, you have to restart or reload the service? You don't want to restart each service every time one of the multiple files changes, so you end up having to build a queue to handle the service notifications.

Say I've got a few dozen users on my hosts and someone leaves. Which is easier - writing a purge script which knows how to purge the user from all environments that you support, or this...

    user { 'joe' : 
      ...
      ensure => absent
      ...
    }
I'm certain that that you can do these things with fabric scripts, but by the time you have, you've either written your own configuration management tool (which is unlikely to be as robust as chef or puppet) or you have a hodgepodge of unmaintainable scripts.

Re: How I spend my first 5 minutes on a server

#237
post #221

Earlier quoted context omitted.

If someone logs in directly as root, how do you track root access? By their SSH-key and IP address (both of which are logged to a remote syslog-server). Given that you should be discouraging root use and using sudo or scripts to automate admin tasks, why are you encouraging admins to log on as root? That is a false premise. I'd never encourage anyone to use "sudo" because it's a source of errors; people get the escap…

Logging the ssh key used sounds interesting - not something I've tried before. I think you're doing something wrong with sudo. I'm not talking about "reentering your password" in the Ubuntu or Mac sense. I'm talking about giving access to specific limited commands. And what distro are you running where local privilege escalation is common?

I'm talking about giving access to specific limited commands.

By definition the set is either so limited as to be essentially useless, or it opens (usually multiple) straightforward paths to a root-shell. Pretty much all sudoers-files I have seen fall in the latter category ("but he can only write the apache config-file!").

And what distro are you running where local privilege escalation is common?

https://www.google.com/search?q=linux+local+privilege+escala...

Re: How I spend my first 5 minutes on a server

#238
post #177
post #165

Earlier quoted context omitted.

If they have had access to a shared account, how much work are you willing to put in to verify that there's nothing in that shared account that will get executed by another user later that will quietly reinstate a key? A reason to have separate accounts is that not only do you terminate access, you also have an easier time ensuring that less of what that person had access to could have been compromised. (This of cour…

But if you give people any access to any system surely this is a concern. As a software dev, maybe I've inserted something into one of the build scripts that quietly re-opens my backdoor to the source control server... I haven't, but if you assume actually malicious users you're probably going to end up with something so locked down it's useless. Aren't you?

It's commonly stated that 9 out of 10 security threats come from employees or other insiders. You should assume malicious employees. Sooner or later you will hire the wrong person.

Now, you must also have a functioning system, and so you may take risks by leaving things more open than you would like if you don't have the resources to thoroughly lock everything down.

But wherever locking things down further costs you very little, you should take the opportunity. And elsewhere you should asses what level of protection you can afford. Ultimately it is a cost-benefit analysis. Many risks are not worth spending money protecting against. Others are vital.

But even disregarding malicious users: Individual user accounts is not just a protection against malicious users, but against careless users. When someone sets a password that gets guessed, you want to be in a position where exploiting that persons credentials is as hard as possible, and tracking down actions taken via the account is as easy as possible.

And yes, you could insert something into a build script. But if the build script is committed, and the commit was pushed from a named, individual account, you're now at the risk of going to jail. Creating deterrents is often a sufficient risk mitigation strategy to be acceptable.

Re: How I spend my first 5 minutes on a server

#239
post #223

Earlier quoted context omitted.

Quite the contrary: You're probably going to end up with documented procedures for deploying software that are simplified, follow existing standards/best practices, and don't rely on complex stone soup build/init scripts concocted by inexperienced developers (and I've seen some doozies). A developer is more likely to create better and more easily maintainable software if the target audience is assumed to be an ordina…

Not quite sure I understand what you're saying here, I'm not talking about the software being produced, but the systems used to produce it. What I was trying to say was that there's not really any way for you (server admin guy) to know if I (software dev guy) have inserted something malicious into a script that all the other software folks run constantly (software build system, NOT server build/init script, NOT deplo…

I see, but I think the same principle applies, even in this narrow case. As a server admin guy or fellow software dev guy, I have to trust that any code you've written has been properly reviewed before checking it into a repository that I pull from. Fortunately, version control tools make this trivial, but you're right, the policy and infrastructure supporting it has to be in place, otherwise you're depending only on voluntary peer review.

Note that as an attacker, there's a high risk of exposure and identification in the scenario you describe, and that's a good thing. A well secured system shouldn't merely prevent attacks, it should also protect innocent users from suspicion (another reason why shared accounts are discouraged).

Re: How I spend my first 5 minutes on a server

#240

Beginner or not, you should probably use visudo [1] instead of vim /etc/sudoers for the sanity checks that it provides, if nothing else. A botched edit of /etc/sudoers that locks you (along with every other user) out of administrative access is an unpleasant way to learn this. [1] http://linux.die.net/man/8/visudo

Sure, as long as you remember to use

    EDITOR=emacs visudo
;-)
Post reply on HN