Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

111–120 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#111

Post by papito »

c2h5oh wrote: post #87
lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

LOL. Priceless. Having these skills is very valuable. Us old farts used to do a lot with what today would be called "bootstrapped". Scarcity is no longer a "problem", except that it is. Scarcity keeps you sharp, efficient, on the edge - where you need to be. It's also - cheaper.

Re: Reclaiming the lost art of Linux server administration

#112

Post by jenscow »

johnchristopher wrote: post #97
jenscow wrote: post #78

Earlier quoted context omitted.

Yes, well if a script I wrote has somehow ended up on a machine without bash, then I'd be more worried about other assumptions the script makes.

That's missing the point. Some server I know don't have vim. Traefik docker image is running ash and not bash. Tomcat image hasn't vim. Etc. /bin/sh is there. No worry about assumptions. No bashism, no fish, no zsh.

That's fine. My scripts tend to run on a single machine.. otherwise, probably the same/similar Linux distro.

So for me, if there's not even bash then I've also surely not accounted for other peculiarities on the system.

Re: Reclaiming the lost art of Linux server administration

#113

Post by zikduruqe »

Yuioup wrote: post #86
lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

Yes but you have talent and a lifetime of experience, plus space for a noisy 42u rack full of servers, but not everybody does...

And a tolerant wife/spouse.

My closet of RPi's are quiet.

Re: Reclaiming the lost art of Linux server administration

#114

Post by mark242 »

jimmaswell wrote: post #28

From my experience I would never recommend giving up control of your servers to some third party. Weeks wasted waiting for useless support teams to get back to you on something you could have fixed in 10 minutes if you had root. Opaque configuration issues you can't debug without said useless support team. Needing permission and approval for every little thing on prod. If I was ever at a high level in a company I'd n…

> If I was ever at a high level in a company I'd never go farther than some AWS load balancing or whatever on cloud instances we still have root on.

Your competitors would salivate at this statement, fyi. Speed is a competitive advantage. AWS is not "let's rent a big ball of EC2 servers and call it a day", and anyone who treats it like that is going to get eaten alive. If you have not looked at -- for example -- Dynamo, you should. If you have not looked at SQS, you should. The ability to have predictable, scalable services for your engineers to use and deploy against is like dumping kerosene onto a fire, it unlocks abilities and velocity that more traditional software dev shops just can't compete against.

Re: Reclaiming the lost art of Linux server administration

#115

Post by Godel_unicode »

rank0 wrote: post #44
warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

Do you have a static IP? I have a homelab too but getting “enterprise grade” service from comcast seems to be my biggest barrier to scaling without leaning on aws.

Rent a $5 VPS, run a VPN tunnel from your lab to that box, and run a reverse proxy on it. You'll get some additional latency, but that's about it.

Re: Reclaiming the lost art of Linux server administration

#116

Post by notyourday »

candiddevmike wrote: post #13

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

> Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

You don't need pet servers. Puppet or Ansible make your baremetal cattle.

Re: Reclaiming the lost art of Linux server administration

#117

Post by notyourday »

chasil wrote: post #3

-"As for scripting, commit to getting good at Bash." That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. This does not implement much of bash and will fail spectacularly on the simplest of scripts. This is also an issue on systems using Busybox. A useful approach to scripting is to grasp the POSIX shell first, then facets of bash and Korn as they are needed. -"As a practi…

> That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh.

#!/bin/bash

There, I fixed your "what shell is /bin/sh" problem.

Re: Reclaiming the lost art of Linux server administration

#118

Post by candiddevmike »

notyourday wrote: post #116
candiddevmike wrote: post #13

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

> Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers. You don't need pet servers. Puppet or Ansible make your baremetal cattle.

I think most folks would argue "cattle" means using imaging to manage/replace your fleet. Using something like puppet or ansible against a fresh install implies a level of individualism towards each system as they "may" have minute details based on when puppet/ansible ran, even if they're part of a dynamic inventory of some sort.

Re: Reclaiming the lost art of Linux server administration

#119

Post by akireu »

c2h5oh wrote: post #87
lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

It's 2022 and we're about to rediscover something we know for 40 years already: mainframes are freaking expensive.

Re: Reclaiming the lost art of Linux server administration

#120

Post by candiddevmike »

0xCMP wrote: post #94
candiddevmike wrote: post #13

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

In my experience Pet servers are a good starting point (you really should _graduate_ from Pet servers into all the various immutable/cattle stuff), but it can quickly require discipline from the Admins. They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutab…

Sounds like you need a new CTO.
Post reply on HN