Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 68 posts

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#31

Post by matt_wulfeck »

cat199 wrote: post #8

anyone have any pointers for simple an API driven managment of DNS/DHCP? (like, I don't want to have to configure 1000 moving parts) typically this seems to fall into the 'roll your own' or 'giant lumbering enterprise behemoth' category that does 10 other things. I'm looking for the sweet spot.

At any reasonable scale you typically wouldn’t use plain DNS if you have to do that kind of figuration. It would be done with a service discovery service which handles SRV records.

That being said route53 has a reasonable management API.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#32

Post by antoncohen »

zie wrote: post #28

I'm going to mostly disagree with everyone here, much to my karma's detriment ;P I agree the end-goal should be infrastructure as code, and everyone here has covered those tools well. You also want monitoring across your infrastructure. Prometheus is the new poster-boy here, but the Nagios family, and many other decent OSS solutions exist as well. But you still need documentation. Your documentation should exist wher…

I totally agree, but having "infrastructure as code" means less documentation.

Instead of documenting all the commands involved in configuring a machine as service X (ssh, run apt-get, paste this, etc.), I have documentation on how work with the configuration management system (roles in the roles/ directory, each node gets one role, commit to git, open PR, etc.). That documentation is in .md files in the config management source repo.

Instead of documenting how to rack a server (print and attach label to front and back, plug power into separate PDUs, enter PDU ports into management database, etc.), I document Terraform conventions (use module foo, name it xxx-yyy, tag with zzz, etc.).

It ends up being less documentation, as the "code" serves to document the steps taken, so the documentation can be higher level. Or if it isn't less documentation, it is documentation that needs to be updated less often, so hopefully there will be less drift between docs and what actually exists.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#33

Post by seorphates »

The past decade has been interesting and I'm still processing it.

My current thoughts are that an appropriate approach is for your systems to document themselves via the applications that they run - inside out.

Though I must abide I cannot fully subscribe to "infrastructure as code" anymore. It has proven just another shift, primarily in toolsets and who (or what) gets say and sway over the capacity, capabilities and efficiencies of the thing you actually care about - the app stack and all of its assembled functionality.

In other words most approaches are still "outside in" - one defines 'x' for deploy fitments and that typically over and over and over again and, typically, with a rigidity that can too easily override and overrule effectively caging your application in scale and scope. With my current tact I am trying to provide for 'y' to "self identify" (via some/any form of config mgmt) where from here you can begin to effectively "deploy to any" by hooking the "application config as code" that, in turn, defines its infrastructure and deploys "outward". The "infrastructure as code" then becomes the servant with its objects and platform definitions etc. and the "appconfig as code" becomes the master where the latter defines its own scope and scale.

Infrastructures have a funny way of mutating into inefficient "definitions" of something that once made sense, on the first day, and forevermore complicating progress with capacity, rules and opinions.

But, generically, snmp is still pretty cool for telling me what I need to know. Strapped that into any end engine and, boom, ask any question, request any inventory.

So.. I track apps, not systems. Systems are expendable, applications are not.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#34

Post by hobofan »

antoncohen wrote: post #32
zie wrote: post #28

I'm going to mostly disagree with everyone here, much to my karma's detriment ;P I agree the end-goal should be infrastructure as code, and everyone here has covered those tools well. You also want monitoring across your infrastructure. Prometheus is the new poster-boy here, but the Nagios family, and many other decent OSS solutions exist as well. But you still need documentation. Your documentation should exist wher…

I totally agree, but having "infrastructure as code" means less documentation. Instead of documenting all the commands involved in configuring a machine as service X (ssh, run apt-get, paste this, etc.), I have documentation on how work with the configuration management system (roles in the roles/ directory, each node gets one role, commit to git, open PR, etc.). That documentation is in .md files in the config manag…

Ah the good old "self-explanatory code that needs no documentation".

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#35

Post by zie »

antoncohen wrote: post #32
zie wrote: post #28

I'm going to mostly disagree with everyone here, much to my karma's detriment ;P I agree the end-goal should be infrastructure as code, and everyone here has covered those tools well. You also want monitoring across your infrastructure. Prometheus is the new poster-boy here, but the Nagios family, and many other decent OSS solutions exist as well. But you still need documentation. Your documentation should exist wher…

I totally agree, but having "infrastructure as code" means less documentation. Instead of documenting all the commands involved in configuring a machine as service X (ssh, run apt-get, paste this, etc.), I have documentation on how work with the configuration management system (roles in the roles/ directory, each node gets one role, commit to git, open PR, etc.). That documentation is in .md files in the config manag…

Yes, I didn't cover what goes into the documentation, as that is mostly site-specific, but I mostly agree with you... mostly. Instead of documenting run apt-get, ssh, etc to start up service X, now you have to document how your tools are setup, Ansible, Terraform, etc. Plus your code needs documentation about why it's setup the way it is.

You still need high-level stuff, policies, etc. Security guides, none of this has changed.

You also have to document your snowflakes, how you handle the wacky snowflakes, why they exist, etc.

Ideally your documentation should be such that it would pass the hit-by-a-bus test. I.e. if you or your entire team got hit by a bus, someone with a clue could come in, read your documentation and continue.

My docs are not at that stage, but every time I mess about with something I try to read through the docs attached, and verify and add to them, so that hopefully someday we will get there.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#36

Post by toomuchtodo »

antoncohen wrote: post #32
zie wrote: post #28

I'm going to mostly disagree with everyone here, much to my karma's detriment ;P I agree the end-goal should be infrastructure as code, and everyone here has covered those tools well. You also want monitoring across your infrastructure. Prometheus is the new poster-boy here, but the Nagios family, and many other decent OSS solutions exist as well. But you still need documentation. Your documentation should exist wher…

I totally agree, but having "infrastructure as code" means less documentation. Instead of documenting all the commands involved in configuring a machine as service X (ssh, run apt-get, paste this, etc.), I have documentation on how work with the configuration management system (roles in the roles/ directory, each node gets one role, commit to git, open PR, etc.). That documentation is in .md files in the config manag…

Sit down with another sysadmin and have them go through your Terraform repo; if they have to ask more than 3 times why something is done a certain way, your "infrastructure as code" as documentation is insufficient.

Source: 16 years in various ops roles

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#37

Post by atsaloli »

As a professional sysadmin, my go to reference on this is "Documentation Writing for System Administrators", from the Short Topics in System Administration series.

https://www.usenix.org/short-topics/documentation-writing-sy...

Also, this talk was very good:

https://www.usenix.org/legacy/event/lisa08/tech/gelb_talk.pd...

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#38

Post by FatalBaboon »

Like many here, I keep it described in ansible and documentation inside a git repository.

But I feel like it's lacking. After a while you have so many ansible playbooks and roles that they cannot give you a birds-eye view anymore.

I think I would MUCH prefer to have some sort of HTML representation, where adding an instance/service starts by adding to that representation, and you could click on every link or node to show its golden image setup, ansible configuration, etc.

THAT, I could show to a newcomer and he'd get it.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#39

Post by petepete »

FatalBaboon wrote: post #38

Like many here, I keep it described in ansible and documentation inside a git repository. But I feel like it's lacking. After a while you have so many ansible playbooks and roles that they cannot give you a birds-eye view anymore. I think I would MUCH prefer to have some sort of HTML representation, where adding an instance/service starts by adding to that representation, and you could click on every link or node to…

I'm no expert but doesn't Ansible Tower do that?

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#40

Post by FatalBaboon »

petepete wrote: post #39
FatalBaboon wrote: post #38

Like many here, I keep it described in ansible and documentation inside a git repository. But I feel like it's lacking. After a while you have so many ansible playbooks and roles that they cannot give you a birds-eye view anymore. I think I would MUCH prefer to have some sort of HTML representation, where adding an instance/service starts by adding to that representation, and you could click on every link or node to…

I'm no expert but doesn't Ansible Tower do that?

Ansible Tower lets you execute a playbook via a web GUI, and keeps a log of who executed what.

I'm not sure if it also shows some infrastructure graphs, but I'm talking about knowing if links are up, how they are firewalled, where the config for each thing is, etc.

When you host tens of services on hundreds of machines, this information is hard to get a grasp on, no matter what you do or how well you documented everything, because it takes a while to read through it.

Post reply on HN