Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Old school Linux administration – my next homelab generation

scholz.ruhr

Post reply on HN
Print view View original

31–40 of 115 posts

Re: Old school Linux administration – my next homelab generation

#31

Post by bayindirh »

As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up.

A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel).

Also, not installing a n+3 Kubernetes cluster with an external storage backend reduces overheads and number of running cogs drastically.

VMs, containers, K8S and other things are nice, but pulling the trigger so blindly assuming every new technology is a silver bullet to all problems is just not right on many levels.

As for home hardware, I'm running a single OrangePi Zero with DNS and SyncThing. That fits the bill, for now. Fitting into smallest hardware possible is also pretty fun.

Re: Old school Linux administration – my next homelab generation

#32

Post by Sebb767 »

don-code wrote: post #4

My home lab sounds pretty similar to the author's - three compute nodes running Debian, and a single storage node (single point of failure, yes!) running TrueNAS Core. I was initially pretty apprehensive about running Kubernetes on the compute nodes, my workloads all being special snowflakes and all. I looked at off-the-shelf apps like mailu for hosting my mail system, for instance, but I have some really bizarre pos…

I fully agree. If you come from "old-school" administration, Docker and Kubernetes seem like massive black boxes that replace all your known configuration screws with fancy cloud terms. But once you get to know them, it just makes sense. Backups get a lot simpler, restoring state is easy and keeping things separated just becomes a lot easier.

That being said, I can only encourage the author with this plan. All those abstractions are great, but at least for me it was massively valuable to know what you are replacing and what an old-school setup is actually capable of.

Re: Old school Linux administration – my next homelab generation

#34

Post by jbotdev »

blinkingled wrote: post #11

It's absolutely fine and I'd argue essential to know Linux system administration even for people who mostly work in the cloud. From my recent experience sys admin skills seem to be a lost art - people no longer have to care about the underpinnings of a Linux system and when problems hit it shows. It's interesting from a job market standpoint how little incentive there is to learn the basics when more and more we are…

I think nowadays requiring that all application developers should also do ops (DevOps?) is a bad idea. Sure they should have basic shell skills, but when you’re on Kubernetes or similar, understanding what’s underneath is not vital. Instead, rely on specialized teams that actually want to know this stuff, and become the experts you escalate to only when things really go sideways and the abstractions fail (which is rare if you do it well). If your budget is too small for this, there are always support contracts.

As someone who’s been hiring for both sides, I see this reflected in candidates more and more. The good devs rarely know ops, and the good ops rarely code well. For our “platform” teams, we end up just hiring good devs and teaching them ops. I think the people that are really good at both often end up working at the actual cloud providers or DevOps startups.

Re: Old school Linux administration – my next homelab generation

#35

Post by irusensei »

I think homeserver and home lab are different things.

For homeserver I use some rock chip arm computers. It runs stuff that is (arguably) useful like a file server and a local DNS with ad and malware filtering rules. These are two fanless ARM machines that use less than 15W idle and they have a single OS and funny names. You can even say 2 machines is a bit too much already.

For home lab I’ve set a k3s environment on Oracle free tier cloud. No personal data or anything that would cause me problems if Oracle decides supporting my freeloading ass is not worth the hassle. I can test things there for exemple recently dabbling into Prometheus since workplace will soon introduce it as a part of a managed offering for Azure Kubernetes services.

Re: Old school Linux administration – my next homelab generation

#36

Post by marginalia_nu »

neoromantique wrote: post #30
marginalia_nu wrote: post #26

Earlier quoted context omitted.

> Containers of course add some overhead, however it is negligible in modern world. I feel the people most enthusiastically trying to convince you of this are the infrastructure providers who also coincidentally bill you for every megabyte you use. > I honestly don't know what you have to do to get 20 gigs of overhead with containers, dozens of full ubuntu containers with dev packages and stuff? Maybe 5 Gb from the c…

>I feel the people most enthusiastically trying to convince you of this are the infrastructure providers who also coincidentally bill you for every megabyte you use. They don't though? Cloud providers sell VM tiers, not individual megabites, and even then on Linux there is barely any overhead for anything, but memory, and memory one is, again, if you optimize it far enough is negligible. >but I ran microk8s which was…

> They don't though? Cloud providers sell VM tiers, not individual megabites, and even then on Linux there is barely any overhead for anything, but memory, and memory one is, again, if you optimize it far enough is negligible.

They don't necessarily bill by the RAM, but definitely network and disk I/O, which are also magnified quite significantly. Especially considering you require redundancy/HA when dealing with the distributed computing

(because of that ol' fly in the ointment

   r    n-r    n!
  p (1-p)   -------
            r!(n-r)!
)

> My k3s with a dozen of pods fits in couple gigs.

My Kibana instance alone could be characterized as "a couple of gigs". Though I produce quite a lot of logs. Although that is not a problem with logrotate and grep.

Re: Old school Linux administration – my next homelab generation

#37

Post by irusensei »

bayindirh wrote: post #31

As a person who manages a big fleet of servers containing both pets and cattle, the upkeep of the pets is nowhere near the cloud-lovers drum-up. A server installed with half-decent care can run uninterrupted for a long long time, given minimal maintenance and usual care (update, and reboot if you change the kernel). Also, not installing a n+3 Kubernetes cluster with an external storage backend reduces overheads and n…

Where I work maintaining server is a big pain in the ass because of the ever growing security and regulatory compliance requirements. The rules makes patching things an exercise in red tape frustration.

Last year when we’ve been asked to redeploy our servers because the OS version was being added to a nope list we decided to migrate to Kubernetes so we don’t have to manage servers anymore (the nodes are a control plane thing so not our problem). Now we just build our stuff using wathever curated image is there that can be used and ship it without worrying about OS patches.

Re: Old school Linux administration – my next homelab generation

#38

Post by neoromantique »

marginalia_nu wrote: post #36
neoromantique wrote: post #30

Earlier quoted context omitted.

>I feel the people most enthusiastically trying to convince you of this are the infrastructure providers who also coincidentally bill you for every megabyte you use. They don't though? Cloud providers sell VM tiers, not individual megabites, and even then on Linux there is barely any overhead for anything, but memory, and memory one is, again, if you optimize it far enough is negligible. >but I ran microk8s which was…

> They don't though? Cloud providers sell VM tiers, not individual megabites, and even then on Linux there is barely any overhead for anything, but memory, and memory one is, again, if you optimize it far enough is negligible. They don't necessarily bill by the RAM, but definitely network and disk I/O, which are also magnified quite significantly. Especially considering you require redundancy/HA when dealing with the…

>They don't necessarily bill by the RAM, but definitely network and disk I/O, which are also magnified quite significantly. Especially considering you require redundancy/HA when dealing with the distributed computing

Network and Disk I/O come with basically 0 overhead with containers.

> Especially considering you require redundancy/HA when dealing with the distributed computing

Why? This is not an apples to apples to comparison then. If you don't need HA, just run a single container, there's plenty other benefits besides easier clustering.

>My Kibana instance alone could be characterized as "a couple of gigs". Though I produce quite a lot of logs. Although that is not a problem with logrotate and grep.

How is Kibana(!) relevant for container resource usage? Just don't use it and go with logrotate and grep? Even if you decide to go with a cluster, you can continue to use syslog and aggregate it :shrug:

Re: Old school Linux administration – my next homelab generation

#39

Post by neoromantique »

blueflow wrote: post #24
neoromantique wrote: post #18

Earlier quoted context omitted.

If you only run your own software and are conscious about your dependencies, then dependency creep is gonna be less of an issue, but don't forget that your dependencies often bring theirs too. And third-party software is very often much less mindful of it. But even besides dependency creep, containers simplify your software env, simplify backups, migrations, permissions(although container are not replacement for secu…

It makes it easier in the sense of "better accessible", not simpler. The docker network setup and namespace management are actually quite complex.

Docker is not the only option when it comes to containers

Re: Old school Linux administration – my next homelab generation

#40

Post by amtamt »

wjnc wrote: post #19
blinkingled wrote: post #11

It's absolutely fine and I'd argue essential to know Linux system administration even for people who mostly work in the cloud. From my recent experience sys admin skills seem to be a lost art - people no longer have to care about the underpinnings of a Linux system and when problems hit it shows. It's interesting from a job market standpoint how little incentive there is to learn the basics when more and more we are…

I see this in my firm. Non-IT management loves to resort to the lowest denominator of 'systems', that is - systems built by people who are not trained as administrators nor experienced, but mainly follow online guides /and deliver/ (at p50). When I mention the resilience of the systems we maintain to senior directors the most common reply is that 1. the activities are not production and 2. the cost of letting our pro…

So many people somehow believe a single VM with no disk snapshot/ back-up, having a floating IP, running ad hoc scripts to bring up production workload is "production quality", only because it is running in cloud.
Post reply on HN