Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

121–130 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#121
I would argue to you don't even need to put that much effort into learning bash scripting, you can totally get away with knowing systemd, journalctl, nginx, apt, ssh and docker and how to run them through bash.

Everything else is per-software files configuration and running commands from the software setup documentation.

Plus, I would run a server with a DE simply because I want to be able to look into databases with a GUI and do config files editing with a nice text editor.

Re: Reclaiming the lost art of Linux server administration

#122

Earlier quoted context omitted.

> Entirely eliminating "system administration" from my duties... ... and adding "cloud administration". What is it with people doing completely one-sided analysis even when they experiment the thing by themselves? Is cloud administration less time consuming than system administration? That's not my experience, so I'm quite interested on how it got so.

> Is cloud administration less time consuming than system administration? Infinitely, and if you look at it from a startup lens it only makes sense. One needs to point only at the recent log4j incident. This is obviously a gigantic black swan event, but even just ongoing security patching at the OS level can be a full-time gig. There is absolutely no substitution for being able to ship code to a platform that just ru…

> in the future, 100% of the code that you write will be business logic

The present reality of Lambda is quite different though. Even though the code of the function itself is more or less "business logic" (although this is a really meaningless term when we're talking about known programming languages and computers), the scaffolding around it with Terraform/CloudFormation/Serverless/etc. is substantial, riddled with quirks and is really time-consuming to figure out and update. I don't think I spend less time on this accidental complexity now when we have most of our logic in Lambda, compared to the times when we were just running Flask apps in a VM.

This is not to mention how hard one has to fight to overcome the limitations of the runtime, e.g. adding some "warmers" scripts to reduce cold-start latency (no, provisioned concurrency doesn't help and is ridiculously expensive). And then comes the bill after you accidentally created invocation loop between two functions.

Re: Reclaiming the lost art of Linux server administration

#123
post #86

Earlier quoted context omitted.

Yes but you have talent and a lifetime of experience, plus space for a noisy 42u rack full of servers, but not everybody does...

You don't need a 42u rack. You can run a cluster of Raspberry Pi's hidden in your basement ceiling rafters like me.

Short throw from that to this classic:

> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.

http://www.bash.org/?5273

Re: Reclaiming the lost art of Linux server administration

#124
post #86

Earlier quoted context omitted.

Yes but you have talent and a lifetime of experience, plus space for a noisy 42u rack full of servers, but not everybody does...

You don't need a 42u rack. You can run a cluster of Raspberry Pi's hidden in your basement ceiling rafters like me.

Racks are not that expensive, and are a very good way to keep things connected, powered, accessible and tidy.

Heaps of Pis in rafters will quickly turn into a cable spaghetti hell tied into ugly knots.

Re: Reclaiming the lost art of Linux server administration

#125

Earlier quoted context omitted.

> Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers. You don't need pet servers. Puppet or Ansible make your baremetal cattle.

I think most folks would argue "cattle" means using imaging to manage/replace your fleet. Using something like puppet or ansible against a fresh install implies a level of individualism towards each system as they "may" have minute details based on when puppet/ansible ran, even if they're part of a dynamic inventory of some sort.

I'm not following, sorry.

This is cattle:

* PXE boot server(s)

* Image contains masterless puppet bootstrap.

* Server(s) asks git - "give me the bootstrap for my mac address"

* Server(s) gets a list of classes to apply.

* Server(s) applies classes.

Done.

Re: Reclaiming the lost art of Linux server administration

#126
post #22

I used to reach for shell scripts to configure servers, then Puppet, then Salt, and then finally to Ansible. Configuring servers declaratively is such a massive improvement over shell scripts. The fact that Ansible is agentless is also very nice and works very well for when you only have a handful of servers. Only thing I dislike is YML, which I think is yucky!

It's a leaky abstraction, though. The problem is that many systems people that were raised only on these abstractions lack the depth to understand what's under the hood when those other layers do unexpected things.

I dont think there is much abstraction about a apt get command and waiting for exit 0

Re: Reclaiming the lost art of Linux server administration

#127
post #86

Earlier quoted context omitted.

Yes but you have talent and a lifetime of experience, plus space for a noisy 42u rack full of servers, but not everybody does...

People who have been woodworking for decades can own very expensive tools so that they can create very complicated things. People who are experts in cars can own very expensive cars and tools to tune them. People who have been working in music can have very expensive instruments and expensive headphones, microphones, sequencers, etc. We seem to be looking down on experienced "computer experts" and wanting to take the…

I don't think that's the point the commenter was making. The analogous situation would be if someone posted that they made their kitchen table from scratch, and the commenter said that it's great but not everyone has a lathe in their basement, so good that places like Ikea exist as well.

Re: Reclaiming the lost art of Linux server administration

#128
post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

Woah, 42u, that's quite something! How many of those slots are you using? What kind of cooling do you have to have for that closet, and how is the volume for the rest of the house?

Not OP but you find a lot of these kinds of setups on Reddit. Check out /r/homelab if you’re interested. Even crazier is /r/homedatacenter where you’ll find dudes with full tape libraries and everything. Super interesting to browse through.

Re: Reclaiming the lost art of Linux server administration

#129

Earlier quoted context omitted.

> How do you handle traffic spikes, especially from the networking point of view? I don't know about GP but managing your own server doesn't mean you cannot use a CDN with your webapp.

A CDN wouldn't be enough if the traffic spike involves writes.

Oh no a website went down! Oh, wait, that's not an emergency. Where did the idea that every site and service needs five nines availability come from? A side project goes down or is read only for a few hours. Who gives a shit? It's not an insulin pump or nuclear control rods. It's ok for people to be mildly inconvenienced for a short period of time.

Re: Reclaiming the lost art of Linux server administration

#130

Earlier quoted context omitted.

That's when it clicked for me.. comparing my hourly salary rate vs. the cost of running these services "in the cloud." Entirely eliminating "system administration" from my duties was absolutely a net win for me and our team.

Counterintuitively, engineers that run their own servers and infra tend to gain a deeper understanding of what it takes to provide an actual running service to end users. And therefor they write software or at least there is better teamwork with "devops" infra folks. This is off course the highly subjective meaning of a greybeard unixadmin.

I'd say that running infrastructure in the cloud still requires the same deep understanding of what's going on under the hood as running your on-prem infra. A lot of annoying things are taken out: some stuff patches automatically, some other things have an easier updating procedure (and obviously the "physical" aspect is taken care of).. but you still only get the basic elements for a proper infrastructure. Your servers need an environment set up, you need to build a network, add load balancing and replication, monitoring etc. etc..

You can provision some of these things from cloud providers, but your infra is going to go to shit unless you actually understand what they're really providing you and how to use it. If the only thing you can do is upload a docker image to a cloud provider and click the "create server" button, then that's not really infra work at all. It's like Wix for sysadmins.

Post reply on HN