Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Old school Linux administration – my next homelab generation

scholz.ruhr

Post reply on HN
Print view View original

81–90 of 115 posts

Re: Old school Linux administration – my next homelab generation

#81

Post by frellus »

davidwritesbugs wrote: post #78

Can someome explain "Pets" and "Cattle"? Not seen this useage before, think I get it but want to be sure. Pets are servers for love and fun, Cattle are servers exploited for loveless meat & milk?

"pets" - servers which require unique, individual care, love, naming and feeding and are irreplaceable in their function. You know it's a "pet" server when you think long and hard when you name it

"cattle" - servers which have no distinct personality traits, are fed, loved and cared for as a group or heard. You know it's a "cattle" server when the name isn't something you can easily remember

Neither pets nor cattle are "exploited", they all have their jobs to do, however they have an ROI and also a consequence of being unique or not. Cattle tend to be born, graze, die and it's part of the cycle. Pets die, and you are emotionally upset and do something stupid -- and you find yourself doing unholy things to resurrect them like using backup tapes or something

Re: Old school Linux administration – my next homelab generation

#82

Post by jerrac »

I can see the appeal. I've been working on containerizing apps at work. There's a lot of complication there. But I also really really want to be able to run updates whenever I want. Not just during a maintenance window when it's ok for services to be offline. And containers are the most likely route to get me there.

At home, well, I enjoy my work, so I containerize all my home stuff as well. And I have a plan to switch my main linux box to ProxMox and then host both a Nomad cluster and a Rancher cluster.

If I weren't interested in the learning experience, I'd just stick with docker-compose based app deployment and Ansible for configuring the docker host vm.

Re: Old school Linux administration – my next homelab generation

#83

Post by bayindirh »

GordonS wrote: post #75
bayindirh wrote: post #31

As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up. A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). Also, not installing a n+3 Kubernetes cluster with an external storage backend reduces overheads and n…

> A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). For my earlier home setups, this was actually part of the problem! My servers and apps were so zero-touch, that by the time I needed to do anything, I'd forgotten everything about them! Now, I could have meticulously documented everything, but..…

We have found out that, while some applications are installed much easier with Docker, operating them becomes much more harder on the long run.

NextCloud is a prime example. Adding some extensions (apps) on NextCloud becomes almost impossible when installed via Docker.

We have two teams with their own NextCloud installations. One is installed on bare metal, and other one is a Docker setup. The bare metal one is much easier to update, add apps, diagnose and operate in general. Docker installation needed three days of tinkering and headbanging to get what other team has enabled in 25 seconds flat.

To prevent such problems, JS Wiki runs a special container just for update duties for example.

I'd rather live document an installation and have an easier time in the long run, rather than bang my head during some routine update or config change, to be honest.

I document my home installations the same way, too. It creates a great knowledge base in the long run.

Not all applications fit into the scenario I told above, but Docker is not a panacea or a valid reason to not to document something, in my experience and perspective.

Re: Old school Linux administration – my next homelab generation

#84

Post by turtledragonfly »

bayindirh wrote: post #31

As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up. A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). Also, not installing a n+3 Kubernetes cluster with an external storage backend reduces overheads and n…

Amen (:

Checking uptime on my FreeBSD home server ... 388 days. Probably 15-year-old hardware. That thing's a rock.

I can't say I'm very proud of my security posture, though (:

One day I'll do boot-to-ZFS and transubstantiate into the realm of never worrying about failed upgrades again.

Re: Old school Linux administration – my next homelab generation

#85

Post by j45 »

jerrac wrote: post #82

I can see the appeal. I've been working on containerizing apps at work. There's a lot of complication there. But I also really really want to be able to run updates whenever I want. Not just during a maintenance window when it's ok for services to be offline. And containers are the most likely route to get me there. At home, well, I enjoy my work, so I containerize all my home stuff as well. And I have a plan to swit…

Docker especially with Portainer is pretty serviceable for an at home setup.

Once the line has been crossed with "I should be backing this up", "I should have a dev copy vs the one i'm using", the existing setups can port quite well/into something like ProxMox to keep everything in a homelab setup running relatively like an appliance with a minimum of manual system administration, maintenance and upgrading.

Re: Old school Linux administration – my next homelab generation

#86

Post by woopwoop24 »

j45 wrote: post #77
woopwoop24 wrote: post #63

HA is overrated, i'd much rather go for a low mean time to repair. Backups, reinstall, ansible playbook is my way to go if hardware fails, which is quite rare to be honest. HA goes beyond hardware in terms of "electricity, internet connection, storage, location etc.., IMO people quite often underestimate what it means to have real HA --> second location with identical setup to shift workload or even have active-activ…

HA is built into platforms like proxmox, what is the stress?

tl:dr if you need it, go for it :) my 2 cents is not needed at home and even for some of my clients the complexicty vs a simple setup and a few min downtime still speaks for MTTR instead of HA which noone can debug.

not about stress but to have HA proxmox cluster you will need at least 3 machines or fake one with a quorum machine without vms on it. Sure your vms will run if one machines goes down, but do you have HA Storage underneath? ganesha would work but more complexity or another network storage with more machines. Don't get me wrong it is fun to play with, but i doubt any homelab needs HA and cannot have a few minutes downtime. Or what do you do in case your internet provider goes down or when you have a power outage? i don't want to provoke, i have fiber switches and 10G at home and have 2 locations to switch in case one location goes down but i can live with multiple days of downtime if i have to, or if not i take my backups and fire some VM's on some cloudprovider and be back online in a few min and pay for it because backups are in both locations.

i would much rather develop a simple LB + autoscaling group with a deployment pipeline (lambda or some other control loop) and containers on it than a k8s cluster the client is not prepared for. if they outgrow this, most likely the following solution is better than going 100% "cloud" the first time. Most clients go from java 8 Jboss monolith to spring containers in k8s and then wonder why it is such a shitshow. but yeah pays the bills so i am not complaining that often ^^

Re: Old school Linux administration – my next homelab generation

#87

Post by eternityforest »

marginalia_nu wrote: post #22
mt_ wrote: post #6

Automation is useful, even in homelab environments, since manual configuration introduces human errors, and automation can speed up recovery from disasters. This article is just for the sake of it. The author calls it old administration, but he better call it outdated and inefficient Linux administration.

Automation is still subject to human errors, and can propagate them much farther.

Automation makes things repeatable. If errors happen, in the "cattle not pets" mindset, you nuke it from orbit and let Ansible make you a new one from a clean Debian/RHEL image. As long as your backup of user data is good(Better double check that script, if you have a script for that) it doesn't matter.

As long as you avoid technology that takes more than a few minutes to set up, rebuilding is trivial.

Re: Old school Linux administration – my next homelab generation

#88

Post by woopwoop24 »

jerrac wrote: post #82

I can see the appeal. I've been working on containerizing apps at work. There's a lot of complication there. But I also really really want to be able to run updates whenever I want. Not just during a maintenance window when it's ok for services to be offline. And containers are the most likely route to get me there. At home, well, I enjoy my work, so I containerize all my home stuff as well. And I have a plan to swit…

if you have more than one pod running at the same time, otherwise will be downtime if the container spawns somewhere else or am i missing something?

Re: Old school Linux administration – my next homelab generation

#89

Post by dvdkon »

bayindirh wrote: post #83
GordonS wrote: post #75

Earlier quoted context omitted.

> A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). For my earlier home setups, this was actually part of the problem! My servers and apps were so zero-touch, that by the time I needed to do anything, I'd forgotten everything about them! Now, I could have meticulously documented everything, but..…

We have found out that, while some applications are installed much easier with Docker, operating them becomes much more harder on the long run. NextCloud is a prime example. Adding some extensions (apps) on NextCloud becomes almost impossible when installed via Docker. We have two teams with their own NextCloud installations. One is installed on bare metal, and other one is a Docker setup. The bare metal one is much…

I agree some software doesn't fit into Docker-style application containers well, but I'm still a fan of Infrastructure-as-Code. I use Ansible on LXD containers and I'm mostly content. For example, my Flarum forum role installs PHP, Apache and changes system configs, but creating the MySQL database and going through the interactive setup process is documented as a manual task.

I could automate this too, but it's not worth the effort and complexity, and just documenting the first part is about as much effort as actually scripting it. I think it's a reasonable compromise.

Re: Old school Linux administration – my next homelab generation

#90

Post by bravetraveler »

woopwoop24 wrote: post #63

HA is overrated, i'd much rather go for a low mean time to repair. Backups, reinstall, ansible playbook is my way to go if hardware fails, which is quite rare to be honest. HA goes beyond hardware in terms of "electricity, internet connection, storage, location etc.., IMO people quite often underestimate what it means to have real HA --> second location with identical setup to shift workload or even have active-activ…

I completely agree, but there's an important balance to consider. HA is a good investment for things that are 'stateless' -- eg: VIPs/LBs

Rebuilding existing systems (or adding new ones) is excellent, but sometimes life really gets simpler when you have one well-known/HA endpoint to use in a given context

Post reply on HN