Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 68 posts

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#41

Post by linsomniac »

FatalBaboon wrote: post #38

Like many here, I keep it described in ansible and documentation inside a git repository. But I feel like it's lacking. After a while you have so many ansible playbooks and roles that they cannot give you a birds-eye view anymore. I think I would MUCH prefer to have some sort of HTML representation, where adding an instance/service starts by adding to that representation, and you could click on every link or node to…

We are using Rundeck to get that. I spent a year or more basically being the only one in the company who could do Ansible runs, for various reasons, and Rundeck is the web UI we ended up setting up for the other part-time ops people and devs to use. I also looked at stackstorm, but it's free version didn't have user accounts which made it a non-starter for me. Plus, you really only interacted with it via chat, not a web page.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#43

Post by outworlder »

Spinning up new infra: Jenkins crafts Terraform tfvars based on user input, runs plan, asks for confirmation, applies. Terraform state and vars saved to S3. Chef and Ansible for provisioning.

"Documentation", in terms of where stuff is deployed and what is deployed is not really necessary. We save this data to a DynamoDB table, query-able by AWS Lambda functions, so other automation can pick it up and devops can query data.

Documentation on how things work comes from dev teams, on how things are deployed indeed comes from us, just simple wiki pages.

Services running in Kubernetes, K8s worker instances in auto-scaling groups. If one node dies it is killed and brought up, K8s will reschedule the pods. Same for the pods themselves.

Monitoring through Nagios(getting phased out finally), NewRelic and Prometheus. Basic ELK stack for centralized logs.

Thinking about rolling out Vault for credential management. Chatops on the pipeline (getting pieces in place first, like the db mentioned earlier)

I'm trying to get the company on board on immutable infrastructure, but it is proving difficult.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#44

Post by cik »

woodrowbarlow wrote: post #26
cik wrote: post #21

We use Collins ( https://tumblr.github.io/collins/ ) as a Configuration Management Database, Ansible ( https://www.ansible.com/ ) for automation, Terraform ( https://www.terraform.io/ ) + a bunch of homebrew for orchestration, Packet ( https://www.packer.io/ ) for multi-cloud (and hypervisor) image creation and maintenance, powered by Ansible. Every since thing is committed to a series of bitbucket ( https://www.bitb…

what do you mean by "passing a yank test"? i assume "yank test" refers to unplugging the network cable abruptly from the server under test, but what exactly are you looking for when you do that?

A yank test on process and infrastructure is more than a 'did it come up'. It's a "if we totally nuke the thing" - say, were we to rip the hard drives out of a server, fry it, and recreate it - does it come up identicall(is).

That way we know our CMDB is accurate, our workflows are accurate, credentials, ansible, terraform, images, etc. Right down to tickets.

It's how we manage all of our cloud customers.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#46

Post by forgottenpass »

jschwartzi wrote: post #14
tootie wrote: post #11

Can I piggyback and ask how people keep track of deployed software? Like if I have 50 products deployed some of which haven't been touched in 10 years and I want to be able to ramp up a developer to fix a bug on any of them?

In the medical device industry you keep what's called a device history file which tracks the configuration of each device you've sold by serial number. This DHF is meticulously updated whenever something is changed. If someone reports an issue this is information you can use to scope your initial reproduction.

I think you're mixing up DHF and DHR. Design history file, device history record.

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#47

Post by erric »

cat199 wrote: post #8

anyone have any pointers for simple an API driven managment of DNS/DHCP? (like, I don't want to have to configure 1000 moving parts) typically this seems to fall into the 'roll your own' or 'giant lumbering enterprise behemoth' category that does 10 other things. I'm looking for the sweet spot.

PowerDNS has served me well and has a terraform provider. Many of the DNS SaaS offerings are also have decent APIs as does AWS route53

Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?

#50

Post by tyingq »

Vmware's tagging support is a lighter, more realistic option vs a "CMDB".

Come up with a key/value strategy that covers your need to track things like app name, app category, environment (test, dev, load testing, prod, prod/dmz, etc), and it becomes actually usable and up to date versus an always out-of-date CMDB. And it's compatible with cloud resource tagging.

Sometimes, less is more.

Post reply on HN