Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How I spend my first 5 minutes on a server

plusbryan.com

Post reply on HN
Print view View original

181–190 of 355 posts

Re: How I spend my first 5 minutes on a server

#181

Post by RobAley »

jiggy2011 wrote: post #97

The guide recommends blocking SSH access to anything other than your own IP address. The problem is that my IP number sometimes changes at which point I end up locked out totally. So to get around this you either have to allow SSH from anywhere or you have to use some remote KVM system. Most of the remote KVM systems seem to be based on Java applets which is not really something you want to enable on your system. So…

I've never tried this, but could you enable access from a given hostname rather than IP address, and then use a dynamic DNS system like http://www.noip.com/ to automatically update a hostname when your IP address changes? Two downsides I can see are that a) you would be vulnerable to DNS poisoning attack on the server side, but that would be the equivalent to just opening it up to all IPs, and b) dynamic DNS systems can take a short while to refresh the DNS record and/or it can get cached by your servers up-stream DNS servers. Some tinkering with your server to make it always use the authoritive DNS servers for your dynamic DNS hostname would help with that.

I'm just thinking out loud, I've never tried any of that!

Re: How I spend my first 5 minutes on a server

#182

Post by matwizzle »

I'd propose using OSSEC over logwatch & fail2ban. Ossec seems to be a bit of an obscure tool, but a thoroughly functional one at that. Logwatch gives a bit too much info at once to interpret properly, while OSSEC will only alert you when something is actually up.

Ossec provides (among some other things):

* Log file monitoring, with severity levels, occurence counting, time-based rules and auto-response. This means (for instance) you get to watch your auth.log for failed login attempts and after 10 failures fire a script to ban him, alert sysop by email or have hubot alert your sysops. Or whatever floats your boat.

* File integrity monitoring. Make sure noone's been mucking around with your files. It has real-time support (through inotify), but if you don't want use that, make sure you store the database it keeps off-server for forensics if need be. Pro-tip: FIM and auto-updates are a tad unnerving.

* It can watch command output. You can use that to make sure the `netstat -tnap` output doesn't change, for instance.

* For larger/more compliant instances, it has a clientserver setup available.

Re: How I spend my first 5 minutes on a server

#183

Post by StavrosK »

vidarh wrote: post #161
raverbashing wrote: post #155

Earlier quoted context omitted.

You are right, this is a better path But sometimes, and especially for servers that will be delivered to the customer afterwards, it's not practical to use a configuration management tool. Also, millions of servers were deployed before Chef/Puppet appeared. You can't tell they did wrong. Also, Chef/Puppet type solutions may be overkill for some tasks, fabric takes care of the easier cases for example.

It's always practical to use a configuration management tool. It's not always practical to use specific tools like Chef/Puppet. Even a shell script that automates your standard install scripts is better than doing it by hand, because they can ensure you don't forget any steps and verify the state afterwards and ensure you don't forget any of the verification steps either.

I just wrote an ansible script to do these sorts of things yesterday. Now, whenever I get a new server, I just run the script with its ip, and boom, provisioned.

Re: How I spend my first 5 minutes on a server

#184

Post by RobAley »

kbuck wrote: post #95

Why install fail2ban? You already have SSH password auth disabled, and you only allow SSH connections from your office. Won't this just risk banning your own office if someone's SSH client is misconfigured?

If helps if a) you've misconfigured your "office only" rules, or b) someone has breached your office network/one of your local machines (but hasn't discovered any of your keys/passwords yet) and is trying to get into the server. It also helps with non-ssh services too. A belt-and-braces approach. If you accidentally lock yourself out using it, use your out-of-band access to re-enable your logins.

Re: How I spend my first 5 minutes on a server

#185

Post by RobAley »

berlinbrown wrote: post #73

What do you about all of those chinese hackers hitting your sshd server. I have 30 different ips and fail2ban doesn't seem to ban them.

Start by changing your SSH port from 22 to something else. It stops 99% of automated attacks. Beware that it is obscurity, not security though. That said, it does help clear some of the "noise" to focus on the other 1% of attacks where the attackers are actually a bit more sophisticated than a simple script.

Re: How I spend my first 5 minutes on a server

#186

Post by hazov »

networked wrote: post #10

Beginner or not, you should probably use visudo [1] instead of vim /etc/sudoers for the sanity checks that it provides, if nothing else. A botched edit of /etc/sudoers that locks you (along with every other user) out of administrative access is an unpleasant way to learn this. [1] http://linux.die.net/man/8/visudo

On debian or ubuntu you can also include a user on the sudo group:

  usermod -a -G sudo username

Re: How I spend my first 5 minutes on a server

#187

Post by kelnos »

raverbashing wrote: post #164
vidarh wrote: post #161

Earlier quoted context omitted.

It's always practical to use a configuration management tool. It's not always practical to use specific tools like Chef/Puppet. Even a shell script that automates your standard install scripts is better than doing it by hand, because they can ensure you don't forget any steps and verify the state afterwards and ensure you don't forget any of the verification steps either.

Ok, great, that's the spirit You can put all his recommendations in a shell script (as the easiest solution) then run it (and if you ever did this more than ONE time you see the value in it)

The problem is that most of the recommendations are bad.

Having an up-to-date system and only accepting security updates is a good policy. Fail2ban is a good tool (but it's a starting point; you should be doing other things to detect suspicious behavior).

The rest is just bad advice.

Having everyone log in using a single user account is a terrible idea. You can't audit who did what, ever. You have to remember to remove people from authorized_keys when they leave, and also make sure that they haven't left themselves a backdoor -- a cron job that reinstates the key, an extra user account, even just changing the root password to something else (how often do you actually check the root password on your boxes?).

User account management is a pain, so that's why we have things like LDAP. Everyone has their own user account. You can audit who does what on every machine, and for stuff that requires root, sudo will log the things people do (of course, if you let people have root shells, that's harder). The only people who get access to a local account (and/or root, but I still think root should just have a random password that no one knows) are a few sysadmins. When someone leaves, you kill their account in the LDAP server.

Even better, if this is a possibility, put up a VPN, and only allow ssh access via the VPN (using a firewall). Tie the VPN login to LDAP (and don't let non-VPN-admins ssh directly to the VPN server), and then you can be sure that without a user account in LDAP, no one can log into your servers.

Blind-updating systems in production is a terrible idea. Things break in the open source world all the time when you do this. Never ever use unattended-upgrades. You just need to be on top of security updates. Period. No excuses.

You should never even have a "my first five minutes on a server" type thing anyway. Rolling out a new server should be fully automatically operationalized. The first time you log into the server, it should be completely ready to go. It should be ready to go without you needing to log into it at all. This takes a small amount of up-front effort, and will pay off immediately when you bring up your second server.

Re: How I spend my first 5 minutes on a server

#188

Post by benedikt »

mryan wrote: post #129
luser001 wrote: post #90

Earlier quoted context omitted.

Why do you say a VPN server is more secure? Which one? I, for one, trust ssh more than any other software wrt security, especially with password login disabled. Disclaimer: I am not a security expert.

I, also, trust SSH more than any other software. But it is still worth adding an additional layer of security in front of SSH to help protect from exploits. Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack. Your servers allow SSH from anywhere on the internet, and are ev…

This is just faulty logic.

openssh is one of the most secure projects. It's developed by the security obsessed (and I mean it in a kind way) folks at OpenBSD.

I, for one, am ready to place for more trust in openssh than in any VPN daemon. The most commonly used ones are propitiatory.

What if there is a 0-day vuln (not exploit) for these VPN daemons? That far more likely. "Securing" ssh with a VPN is just one step beyond of security by obscurity.

If you are afraid of script-kiddies and scanners, let your sshd listen on a non-standard port.

Re: How I spend my first 5 minutes on a server

#189

Post by kelnos »

lmm wrote: post #132
mryan wrote: post #129

Earlier quoted context omitted.

I, also, trust SSH more than any other software. But it is still worth adding an additional layer of security in front of SSH to help protect from exploits. Let's say that, hypothetically, a 0-day exploit was discovered in SSH which allowed remote code execution. A script kiddie begins trawling the internet for publicly accessible SSH servers to attack. Your servers allow SSH from anywhere on the internet, and are ev…

Your outermost server is the one where you should be most worried about having vulnerabilities - if you have a VPN as the outer layer that means the VPN server must be exposed to the public internet, and anyone who compromises it is in a pretty good position. And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.

anyone who compromises [the VPN server] is in a pretty good position.

Sure. But without a VPN, anyone who compromises even one of your other hosts is in the same position. It's a lot easier to audit a single-purpose VPN server for possible security issues than it is to audit all the application code running on the rest of your production systems.

And I'd rate the odds of a 0-day being found at higher for most VPN software than for SSH.

I wouldn't. And even if you're right, getting a VPN login still doesn't get you anywhere. You still have to be able to ssh to the rest of the hosts. That's why we do security in layers.

Re: How I spend my first 5 minutes on a server

#190

Post by cs02rm0 »

stevekemp wrote: post #146
ck2 wrote: post #44

Your first 5 seconds should probably be w last dmesg

Add "free" and "top" there too.

I've had df -h write out to the motd before for clients who insisted on tiny slices of disk, no automated monitoring and no manual checking of disk space even after it chewed itself up.

I'm not sure it helped, but it made me feel better.

Post reply on HN