Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Tips on Sysadmin Job

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Ask HN: Tips on Sysadmin Job

#31

Post by yabones »

28304283409234 wrote: post #12

Initial thoughts of 20 year veteran: - Create a single-source-of-truth you can automate against. Do that now. Use ansible to fetch and create an overview of what your landscape actually is running: which software, which versions. Find out what is going EOL, as this will bite you soon enough. - don't automate anything else until you understand the full context of what you are automating. Automation is abstraction, and…

Thanks for testinfra, awesome link. Integration with Jenkins and/or Nagios makes this super useful for remediation.

Re: Ask HN: Tips on Sysadmin Job

#32

Post by hardwaresofton »

It seems no one suggested this but maybe you could try getting in touch with the person who got fired? Buy them the beverage of their choice and figure out what happened (and if you’re next in X years) let them know the pickle you’re in (and that you don’t blame them of course), and ask for some tips if they are willing to give you some.

You worked together, surely there was some rapport there?

Re: Ask HN: Tips on Sysadmin Job

#33

Post by yabones »

Just want to say, good luck OP. I was in your position about 3 years ago, though with a smaller company at the time. It was the hardest, most stressful experience in my career so far but I've come out the other side with immeasurably greater confidence and skills. This will be the most brutal learning experience of your life. But, if you need to throw in the towel, there's no shame in that at all.

Re: Ask HN: Tips on Sysadmin Job

#34

Post by bierjunge »

ed_elliott_asc wrote: post #30
300 wrote: post #6

Here's what to do in the first few weeks: 1. Require not 1, but 2 people, for hire, right now. It will take a long time until it gets approved, HR does their thing, and you (or someone else) hire people to help you. 2. Document, as fast as you can, all of the assets in your environment. Don't waste time on looking for software. Plain text or spreadsheet will do for start. 3. Identify the most important assets (server…

6. Any critical assets - verify you can restore and get working offsite with 0 access to the current network

7. Have a backup strategy

7.5 Test your backup strategy every now and then (nothing is more painful than data loss and dysfunctional backups)

8. "All users lie, there is no exception". Sad but true, get used to it.

9. Automate your infrastructure. If something goes down, just recreate it with Chef/Ansible/Terraform/etc.

[edit: added no.9]

Re: Ask HN: Tips on Sysadmin Job

#35

Post by yk »

Run!

Or at least go to your manager and make very clear that it is simply not possible to provide anything like 24/7 incident response when you are alone. That is the kind of conversation, where it is just better to get fired rather than not getting your point across.

The basic structure of the admin job is, that there is always more to do and it is very easy to burn out. There is always more to do, it is always important, but trying to chase some ideal of a perfect system will just exhaust you. Furthermore, it is not a big problem to check your monitoring before going to bed or a few times on the weekend, but that limits your rest periods to a few hours at best, you no longer get a full weekend, and that will grind you down over time. So, manage your time and sanity, and importantly also manage the expectations of your boss and co workers, there is only so much you can do when you're alone.

Re: Ask HN: Tips on Sysadmin Job

#36

Post by Ologn »

300 wrote: post #6

Here's what to do in the first few weeks: 1. Require not 1, but 2 people, for hire, right now. It will take a long time until it gets approved, HR does their thing, and you (or someone else) hire people to help you. 2. Document, as fast as you can, all of the assets in your environment. Don't waste time on looking for software. Plain text or spreadsheet will do for start. 3. Identify the most important assets (server…

> Establish some sort of monitoring for the most important assets.

You need this to know if your web sites, servers and services are running, and to be emailed or contacted when not. I am used to Nagios. I understand there is a Nagios fork, Icinga, which I know little about. Some people use Graphite, which I also know little about.

You can also have it check for disks running out, high CPU loads, inodes running out, heavy I/O etc.

Also, you can email etc. alerts, or pay for a company like Amelia to monitor a dashboard and page/call people at night if a server is down, or to even do some simple scripted remediation attempts.

Re: Ask HN: Tips on Sysadmin Job

#37

Post by Ologn »

bierjunge wrote: post #34
ed_elliott_asc wrote: post #30

Earlier quoted context omitted.

6. Any critical assets - verify you can restore and get working offsite with 0 access to the current network

7. Have a backup strategy 7.5 Test your backup strategy every now and then (nothing is more painful than data loss and dysfunctional backups) 8. "All users lie, there is no exception". Sad but true, get used to it. 9. Automate your infrastructure. If something goes down, just recreate it with Chef/Ansible/Terraform/etc. [edit: added no.9]

Also, backups are kind of thankless - no one ever puts backup maintenance on your schedule, but if a machine goes down everyone expects you have a backup from three weeks ago and that they can retrieve all the data from it. You also have to poll people as to what needs to be backed up. You need to do test recoveries. Some backups should be offsite, and the policy around that should be communicated to managers.

Re: Ask HN: Tips on Sysadmin Job

#38

Post by 0xFFFE »

bierjunge wrote: post #34
ed_elliott_asc wrote: post #30

Earlier quoted context omitted.

6. Any critical assets - verify you can restore and get working offsite with 0 access to the current network

7. Have a backup strategy 7.5 Test your backup strategy every now and then (nothing is more painful than data loss and dysfunctional backups) 8. "All users lie, there is no exception". Sad but true, get used to it. 9. Automate your infrastructure. If something goes down, just recreate it with Chef/Ansible/Terraform/etc. [edit: added no.9]

10. Rotate all the passwords immediately since the other guy has left the organization and have a password policy in place.

11. Understand the reason the other guy was fired make sure you are not the one holding the bag.

12. Have a issue tracking system in place, you will thank yourself later.

13. Have a change management policy in place.

14. Before you make any change full understand the consequences and have a tested plan to get back to a known good state.

Re: Ask HN: Tips on Sysadmin Job

#40

Post by denton-scratch »

loriverkutya wrote: post #5

This is not a promotion, this is making you to do something you have no experience with and making you do the job what the fired person done besides yours. Also being responsible for the whole IT infrastructure alone is definitely what I would not do without any experience. If that’s how the company handles this, I would start to look for a new job immediately.

You now have a shitty job!

I enjoy sysadmin, partly because of the variety of problems you are called-on to solve. But the fact is, if you do the job well, and everything works well, nobody gives you any credit. When things go wrong everybody will be looking at you, and if you haven't fixed it within 20 minutes, then you must be incompetent.

So you're the only sysadmin in a multi-campus company? As someone noted up-thread, you can't do it alone. At least, you need someone on each site that can act as remote hands for you, even if that's not their main job. And you need people to delegate grunt work to - repetitive stuff, like building workstations for new hires. That's IT, but it's not sysadmin.

Get some interesting hardware to take home and play with. Stuff like smart switches and firewall boxes (the same gear your employer uses).

GNU/Linux has much better networking tools than Windows; if you're in a Windows shop, get a laptop with Linux on it.

Whenever you do a manual job you've never done before, let part of the job be automating it, so you never have to do it by hand again. Let the automation scripts be the documentation.

Learn to be pessimistic (a common sysadmin trait). If you have bad news, don't ever sugar-coat it. Things might not be as bad as you thought, in which case your reputation is improved. But what people don't want to hear is that the job takes two hours; and then find after two hours that it's a two-day job.

If you can get yourself a full-time assistant/colleague, work as a pair when you can. It reduces stress a lot if you have someone checking your work as you do it (and your decisions). And you end up with two people knowing the systems, not just one.

Get a hardback A4 notebook, and record everything you do.

In theory, you have a lot of power (c.f. BOFH).

http://bofharchive.com/

You know stuff others don't, and you have superuser powers everywhere. Don't abuse it. Be as helpful as you can. People will be nice to you if they know they might need your help one day. The BOFH was a tyrant, and I don't think he'd get far in a modern business (the BOFH worked in a university IT department, and was God; a sysadmin in a modern business can't behave like the BOFH).*

Post reply on HN