As much as people on HN frown upon anything reddit related, the /r/SelfHosted is an amazing source for what the article topic is about. In fact I came across large organizations running the likes of Proxmox (VE), Ceph, Minio, k3s, Graphana, etc., etc., with a lot of info available in this subreddit. Highly recommended.
Old school Linux administration – my next homelab generation
41–50 of 115 posts
Re: Old school Linux administration – my next homelab generation
#42I’ve been there. Many technologies OP listed .. it all becomes a nightmare without proper HA (but who want’s to have _multiple_ servers at home (servers as enterprise HW, not a computer designation). Then you hit an issue with licensing, old HW not being supported by recent releases (looking at you, ESXi), loud and heat producing boxes in your closet... I am grateful for having this awful experience but I moved on as well.
For me, the middle ground seems to be a very plain Debian server, configured with Ansible and all services running in Docker. I consume a few official images and have a (free) pipeline set up to create and maintain some of mine own. I appreciate the flexibility of moving a container over to different HW, if needed, and also the isolation (both security and dependencies).
Going back pure, OS level configuration is always fun - and often necessary to understand how things work. It’s very true that the new cattle methodology takes away any deeper level of troubleshooting and pushes “SRE” towards not only focusing on their app code but also away from understanding the OS mechanics. Instance has died? Replace it. How did it happen? Disk filled up because of wrong configuration? Bad code caused excessive logging? Server was compromised? _They_ will never know.
Re: Old school Linux administration – my next homelab generation
#43I really like this article just for the straightforwardness of the setup. Pets not cattle should be the home server mantra.
My setup is not quite as simple. I have one homeprod server running Proxmox with a number of single task VMs and LXCs. A task, for my purposes, is a set of one or more related services. So I have an internal proxy VM that also runs my dashboard. I have a media VM that runs the *arrs. I have an LXC that runs Jellyfin (GPU pass through is easier with LXC). A VM running Home Assistant OS. Etcetera.
Most of these VMs are running Docker on top of Alpine and a silly container management scheme I've cooked up[1]. I've found this setup really easy to wrap my head around, vs docker swarm or k8s or what have you. I'm even in the process of stripping dokku out of my stack in favor of this setup.
Edit: the homeprod environment also consists of a SFF desktop running VyOS and a number of Dell Wyse 3040 thin clients running the same basic stack as the VMs, most running zwave and zigbee gateways and one more running DHCP and DNS.
Re: Old school Linux administration – my next homelab generation
#44As much as people on HN frown upon anything reddit related, the /r/SelfHosted is an amazing source for what the article topic is about. In fact I came across large organizations running the likes of Proxmox (VE), Ceph, Minio, k3s, Graphana, etc., etc., with a lot of info available in this subreddit. Highly recommended.
In my experience, Hacker News has been one of the places that's unusually sympathetic to Reddit, probably because it originated with YCombinator. If you get the impression that people here generally dislike Reddit you should take that as a a strong indicator that Reddit has actually become a cesspool.
Re: Old school Linux administration – my next homelab generation
#45It's absolutely fine and I'd argue essential to know Linux system administration even for people who mostly work in the cloud. From my recent experience sys admin skills seem to be a lost art - people no longer have to care about the underpinnings of a Linux system and when problems hit it shows. It's interesting from a job market standpoint how little incentive there is to learn the basics when more and more we are…
I see this in my firm. Non-IT management loves to resort to the lowest denominator of 'systems', that is - systems built by people who are not trained as administrators nor experienced, but mainly follow online guides /and deliver/ (at p50). When I mention the resilience of the systems we maintain to senior directors the most common reply is that 1. the activities are not production and 2. the cost of letting our pro…
They most likely resorted to skipping IT involvement because the department is difficult to work with.
"The cloud" is both a technical solution for flexible workload requirements, and a political solution to allow others to continue delivering when IT is quoting them 3 months of "prep work" for setting up a dev environment and 2 to 4 weeks for a single SSL certificate.
As a consultant, I am confronted to hostile IT requirements daily. Oftentimes, departments hiring us end up setting and training their own Ops teams outside of IT. despite leading to incredible redundancy, that's often credited as a huge success for that department, because of the gained flexibility.
Re: Old school Linux administration – my next homelab generation
#46As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up. A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). Also, not installing a n+3 Kubernetes cluster with an external storage backend reduces overheads and n…
The thing that helped me was realizing that for a homelab set up, running without the extra redundancy is fine. Now for me that meant running k8s on a single box because I was specifically trying to get experience with it, and putting the control plane and actual workloads on a single machine simplified the whole thing to the point that it was easy to get up and running; I had gotten bogged down in setting up a full production grade cluster, but that wasn't even remotely needed for what I was doing.
Re: Old school Linux administration – my next homelab generation
#47Are there any good resources for what to mind as a sysadmin? It seems that most resources I find when searching, or that come across my radar (like this article) are focused on mechanism and how-to.
I often feel a bit out of my depth when dealing with sysadmin tasks. I can achieve pretty much any task I set out to, but it is difficult to ascertain if I have done so well, or with obvious gaps, or if I have reinvented some wheels.
Does anyone have a recommendation for in-depth "So you want to be a sysadmin..." content? Whether books, articles, presentations, or any other format.
Re: Old school Linux administration – my next homelab generation
#48I’ve been there. Many technologies OP listed .. it all becomes a nightmare without proper HA (but who want’s to have _multiple_ servers at home (servers as enterprise HW, not a computer designation). Then you hit an issue with licensing, old HW not being supported by recent releases (looking at you, ESXi), loud and heat producing boxes in your closet... I am grateful for having this awful experience but I moved on as…
> cattle methodology takes away any deeper level of troubleshooting and
> pushes “SRE” towards not only focusing on their app code but also away
> from understanding the OS mechanics
I know some places are trying to redefine "SRE" to mean "sysadmin", but that seems silly given that we'd need to invent a new term to mean the people who write and operate low-level infrastructure software that the rest of a modern distributed stack runs on.
Re: Old school Linux administration – my next homelab generation
#49Automation is useful, even in homelab environments, since manual configuration introduces human errors, and automation can speed up recovery from disasters. This article is just for the sake of it. The author calls it old administration, but he better call it outdated and inefficient Linux administration.
Automation is still subject to human errors, and can propagate them much farther.
So do like you would in old-school, have a staging/testing environment.
Nice thing about automation, is you can just rebuild that easily to a clean state, rather than having to remember how staging is different from prod. And as you write your automation and find those human errors, you put it in code so that you don't forget a step. Nothing more frustrating than thinking you've solved the problem in staging only to run a command in prod that breaks things more, and in a different way, because it turns out the fix was a combination of earlier experimentation plus the command you just ran.
Re: Old school Linux administration – my next homelab generation
#50Are there any good resources for what to mind as a sysadmin? It seems that most resources I find when searching, or that come across my radar (like this article) are focused on mechanism and how-to. I often feel a bit out of my depth when dealing with sysadmin tasks. I can achieve pretty much any task I set out to, but it is difficult to ascertain if I have done so well, or with obvious gaps, or if I have reinvented…
Depends on what system you want to administrate. The topic is very broad.
But it never hurts to understand the fundamentals:
- networking (IPv4/IPv6/TCP/UDP/ICMP/various link layer technologies (ethernet, wifi, ppp, USB, ...)... and concepts like VPNs/VLAN/bridging/switching/routing/...)
- protocols (DHCP, DNS, NTP, SMTP, HTTP, TLS, ...)
- chosen OS fundamentals (so on Linux things like processes, memory, storage, process isolation, file access control, sockets, VFS, ...)
- OS tooling (process managers, logging tools, network configuration tools, firewall, NAT, ...)
- how to setup typical services (mail servers, file servers, http servers/proxies, DNS servers, DNS resolvers, ...)
- how to use tools to debug issues (tracing, packet capture, logging, probing, traffic generators, ...)
- how to use tools to monitor performance of the network and hosts
These are just some basics to run a home network or maybe a small business network.