Can I piggyback and ask how people keep track of deployed software? Like if I have 50 products deployed some of which haven't been touched in 10 years and I want to be able to ramp up a developer to fix a bug on any of them?
Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
11–20 of 68 posts
Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
#12GLPI with FusionInventory for IT Asset Management and Knowledge Base.
GitLab for repositories, adhoc documentation via gists and CI/CD.
Nagios for monitoring.
Open to trying other things out if they make sense.
Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
#13I’ve found Zabbix works decently well and also covers monitoring. Zabbix Maps can be nice to visualize the infrastructure: https://www.zabbix.com/documentation/3.4/manual/config/visua...
Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
#14Can I piggyback and ask how people keep track of deployed software? Like if I have 50 products deployed some of which haven't been touched in 10 years and I want to be able to ramp up a developer to fix a bug on any of them?
In the medical device industry you keep what's called a device history file which tracks the configuration of each device you've sold by serial number. This DHF is meticulously updated whenever something is changed. If someone reports an issue this is information you can use to scope your initial reproduction.
Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
#15I don't do devOps but if I did...
Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
#16Terraform + Datadog + Cloudwatch
http://terraform.io/ http://datadoghq.com/ https://aws.amazon.com/cloudwatch/
Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
#17- keep inventory in a DCIM (we use Netbox) - configure everything as code (we use Ansible for the infrastructure up to OS level, Kubernetes w/ Helm for applications), have it read the values from the DCIM so that the DCIM remains the single source of truth (we need to still get better on this part....) Links: https://github.com/digitalocean/netbox https://www.ansible.com https://www.kubernetes.io That's at work. At h…
I cannot comment on the DCIM side, but I agree on the "everything as code" mantra.
For a relatively small setup I chose a combination of Ansible, Kubernetes and Dockerfiles, but probably any combination will do. All these files are stored in a git repo.
Even after months (or years) neglect, I can easily know what I configured (and why!) and update where needed with a minor effort.
Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
#18It might be helpful if described your infrastructure. There is a pretty big difference between managing physical Windows servers in a data center and managing Linux servers all in AWS.
If you are all or mostly cloud, Terraform + config management with a CI pipeline takes care of a lot. Then a wiki that covers "Getting Started" and a few how-to articles.
For physical infra you need the setup for DHCP, updating DNS based on DHCP, PXE boot imaging, IPMI access and configuration, switch and router configuration, what servers are connected to which switch ports, PDU management and monitoring, and on and on and on.
You end up with something like NetBox (https://github.com/digitalocean/netbox) or Collins (https://tumblr.github.io/collins/), plus a bunch of other stuff gluing things together.
Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
#19When I was working as a sysadmin, I kept a spreadsheet. I was told later of a repository of information that supposedly was what my spreadsheet did, but it didn't add anything new and was much harder to keep up.
I built it up using nmap and then shelling into each individual machine and poking around to see what it did. This was back in the days before everything became virtualized, so each machine on the network was likely physical.
I added information by walking the aisles and copying down the rack location of every machine into another page on the spreadsheet. I eventually hooked up a terminal to them all and matched network addresses to physical machines.
Only took a few weeks and when I was done, I knew things about the network that guys who worked at the business for years didn't know.
There's no substitute for the good old-fashioned way.
I liked that job, it was fun.
Re: Ask HN: How do you document and keep tabs on your infrastructure as a sysadmin?
#20I think it depends a lot on the size of your infrastructure. I've used excel docs on a shared drive pretty successfully where there's not much to keep up on and changes are few.
In larger infrastructure setups (small service provider) we used a combination of netboot, SNMP for monitoring with Observium and Nagios for alerting. We were also a big VMware environment, so naturally we had a lot of inventory tracking available through vCenter as well. I found a lot of opposition to Configuration Management, given the lack of comfort with programming of some sysadmins (Windows admins), so that's something to keep in mind as well. I think mixed environments also can be challenging w/infrastructure as code, but I'd be interested to see how others get through that.