Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#21

Post by cik »

We use Collins (https://tumblr.github.io/collins/) as a Configuration Management Database, Ansible (https://www.ansible.com/) for automation, Terraform (https://www.terraform.io/) + a bunch of homebrew for orchestration, Packet (https://www.packer.io/) for multi-cloud (and hypervisor) image creation and maintenance, powered by Ansible. Every since thing is committed to a series of bitbucket (https://www.bitbucket.org) repositories.

We connect Ansible and Collins through ansible-cmdb (https://github.com/fboender/ansible-cmdb), then tie the entire thing to our ticketing systems ServiceNOW (https://www.servicenow.com/) and Jira Service Desk (https://www.atlassian.com/software/jira/service-desk), and finally, ensure we have history tracking with Slack (https://www.slack.com).

As a given, we yank test the entire world. If it doesn't pass a yank, it straight up doesn't exist.

Whether it's bare-metal, virtualized, para-virtualized, dockerized, mixed-mode, or cloud - we 100% do this all the time. There is not a single change across any environment, that isn't fully tracked, fully reproducible, fully auditable, and fully automated.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#22

Post by evangineer »

antoncohen wrote: post #18

It might be helpful if described your infrastructure. There is a pretty big difference between managing physical Windows servers in a data center and managing Linux servers all in AWS. If you are all or mostly cloud, Terraform + config management with a CI pipeline takes care of a lot. Then a wiki that covers "Getting Started" and a few how-to articles. For physical infra you need the setup for DHCP, updating DNS bas…

For future work, I would definitely consider NetBox and Collins as alternative options to GLPI.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#23

Post by pnutjam »

castis wrote: post #3
AdamGibbins wrote: post #2

I'm not sure I understand your question fully? You write documentation, like you do anything. And configure everything with code, so you can go read it (Terraform, Chef/Puppet/Ansible, etc).

OP is probably looking for someone to go a little further into detail on exactly what you just said.

I use SCC (System Configuration Collector) to document our servers. Everything else is just a collection of grep-able text files on our management server. https://sourceforge.net/projects/sysconfcollect/

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#26

Post by woodrowbarlow »

cik wrote: post #21

We use Collins ( https://tumblr.github.io/collins/ ) as a Configuration Management Database, Ansible ( https://www.ansible.com/ ) for automation, Terraform ( https://www.terraform.io/ ) + a bunch of homebrew for orchestration, Packet ( https://www.packer.io/ ) for multi-cloud (and hypervisor) image creation and maintenance, powered by Ansible. Every since thing is committed to a series of bitbucket ( https://www.bitb…

what do you mean by "passing a yank test"? i assume "yank test" refers to unplugging the network cable abruptly from the server under test, but what exactly are you looking for when you do that?

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#28

Post by zie »

I'm going to mostly disagree with everyone here, much to my karma's detriment ;P

I agree the end-goal should be infrastructure as code, and everyone here has covered those tools well. You also want monitoring across your infrastructure. Prometheus is the new poster-boy here, but the Nagios family, and many other decent OSS solutions exist as well.

But you still need documentation. Your documentation should exist wherever you spend most of your time. Some examples:

* If you spend most of your time on a Windows Desktop, doing windows admin type things, then OneNote or some other GUI note-taking/document program makes sense.

* If you spend most of your time in Unix land(linux, BSD, etc) then plain text files on some shared disk somewhere for everyone to get to, makes WAY more sense. Bonus if you put these files in a VCS, and treat it like code, and super bonus if your documentation is just a part of your Infra as code repositories.

* If you spend your time in a web browser, then use a Wiki, like MediaWiki, wikiwiki, etc.

In other words, put your documentation tools right alongside your normal workflow, so you have a decent chance of actually using it, keeping it up to date, and having others on your team(s) also use it.

We put our docs in the repo's right alongside the code that manages the infrastructure.. in plain text. It's versioned. We don't publish it anywhere, it's just in the repo, but then we spend most of our time in editors messing in that repo.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#29

Post by paydro »

We put everything in code. We have several layers, but they if you're new you can start with the lowest level and make your way up to find out how things are provisioned and configured.

We're on AWS so we use cloudformation for provisioning and saltstack (https://saltstack.com/) for configuration management. Cloudformation templates are written using stacker (http://stacker.readthedocs.io/en/stable/). All AWS resources are built by running "stacker build" so nothing is done by hand. We have legacy resources that we're slowly moving over to Cloudformation, but more than 90% of our infrastructure is in code.

On top of cloudformation and salt we built jenkins (CI and docker image creations), spinnaker (deployment pipeline), and kubernetes (deployment target). The jenkins and spinnaker pipelines are also codified in their own respective git repos.

All the repos here have sphinx setup for documentation purposes and the repos tend to crosslink for references.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#30

Post by itomato »

There are several classes of "infrastructure" as a sysadmin; legacy, new and critical.

Legacy stuff is done the old fashioned way - portscans and nmap. If it has an open port, it's presumed to be intentional. If not, it's a target. I've seen some success using tools like Pysa to "blueprint" existing systems into Puppet code. Tools like SystemImager help here, too - enabling P2V and the creation of "file-based images" compatible with version control and able to PXE boot new clones.

New stuff is from-scratch IaC all the way to the metal. Ansible and git submodules help me build "sandwiches".

Critical stuff blurs the lines. The machines, IP addresses, ports and living connectivity can be documented, and "captured" to a limited extent with the manual mapping and Rsync stuff in the Legacy category. Some of this critical stuff is also "new", and is deployed in that fashion.

What about switchgear and Cisco configs? License strings, key management, site-specific patching - all can complicate things.

More important than any of these is the ability for you and those around you to see and manage the systems as they are launched and terminated.

In the old days, I used to use a shell script on a newly-provisioned host to dump all its' details - dmidecode, environment stuff and so on. Those details were pushed back to a common source and were a real benefit in the days before real config management came on the scene. CFEngine was way too complicated and nebulous at the time.

Post reply on HN