Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

291–300 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#291

Post by dragonsky »

senko wrote: post #160
viraptor wrote: post #139

Earlier quoted context omitted.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

> Here's what the bare metal server didn't come with: [bunch of stuff I don't need] Exactly. Imagine paying for all that when all you need is bare metal. Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do. Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy. Buy what you ne…

Your point is well made... The other thing you need to think about is that all those extra services can come with a reliability cost if your provider is not 100%. Most outages we've encountered have been because the HA infrastructure has decided to have problems rather than the underlying hardware or actual service we need to run.

Having all that "best practice" service is great if it works well, but when it becomes a checkbox on the purchase order then it can cause far more problems than it solves.

I have found that a lot of the push to outsource hosting is simply an attempt to deflect responsibility for problems rather than in expectation of actually providing more reliability.

Re: Reclaiming the lost art of Linux server administration

#292

Post by whalesalad »

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

Bonus the servers keep the house warmer in the winter :D

Re: Reclaiming the lost art of Linux server administration

#293

Post by b112 »

outworlder wrote: post #265
senko wrote: post #160

Earlier quoted context omitted.

> Here's what the bare metal server didn't come with: [bunch of stuff I don't need] Exactly. Imagine paying for all that when all you need is bare metal. Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do. Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy. Buy what you ne…

> Exactly. Imagine paying for all that when all you need is bare metal. Yes. The opposite is also true though: Imagine not wanting to pay for that and needing it! There's a reason why most homes connect to the power utility companies. Yes, we can run generators ourselves. Does it make sense to do that? Not usually. Same thing with this server. If it makes sense for your use-case, outstanding. In many cases, people ar…

Exactly! Who needs a kitchen in their home, it's better to just order in all the time. All that complexity of cooking daily, preparing and storing food, cleaning dishes, etc, etc. I mean, who's going to even do the cooking!

See... anyone can pick , and describe how people don't do it. Of course, you cite a generator, others use solar cells, right?

Re: Reclaiming the lost art of Linux server administration

#294

Post by spicybright »

> As a practical goal, you should be able to recreate your host with a single Bash script.

I think this is ideal, but I've yet to be able to do this or see a solid example.

At my old job I had to do exactly this, and it was really hard to get things right.

I'm much more seasoned now, but I still don't think I could do it lol

Re: Reclaiming the lost art of Linux server administration

#295

Post by b112 »

primax wrote: post #290
viraptor wrote: post #139

Earlier quoted context omitted.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

Also physical security, power, physical rack cost, networking, salaries to manage everything, and all those other hidden costs

There are no "hidden costs". They're well known.

Factor them all in, and "the cloud" is 1000s to millions of times more expensive than bare metal. I've seen people pay $100k/month, for services which could run on an $140/month bare metal server failover pair.

Meanwhile, people still spend enormous resources managing "the cloud", writing code to deploy to the cloud, dealing with edge cases in the cloud.

There are no savings, time wise, management wise, or money wise, with the cloud.

You are paying for ignorance.

Re: Reclaiming the lost art of Linux server administration

#296

Post by DenseComet »

spicybright wrote: post #294

> As a practical goal, you should be able to recreate your host with a single Bash script. I think this is ideal, but I've yet to be able to do this or see a solid example. At my old job I had to do exactly this, and it was really hard to get things right. I'm much more seasoned now, but I still don't think I could do it lol

I've done this with Nixos. Bash and most other tools are too brittle to get the system back to the same exact state.

Re: Reclaiming the lost art of Linux server administration

#297

Post by MonaroVXR »

jhickok wrote: post #49
invokestatic wrote: post #8

I used to reach for shell scripts to configure servers, then Puppet, then Salt, and then finally to Ansible. Configuring servers declaratively is such a massive improvement over shell scripts. The fact that Ansible is agentless is also very nice and works very well for when you only have a handful of servers. Only thing I dislike is YML, which I think is yucky!

Out of curiosity, have you tried tools like Pulumi? I've never used it but as a longtime Ansible user it's something that has my interest.

Pulumi isn't for your own infrastructure, but for Cloud providers. I didn't search for a Cloud provider-less Pulumi/Terraform alternative yet.

Re: Reclaiming the lost art of Linux server administration

#298

Post by reaperducer »

senko wrote: post #160
viraptor wrote: post #139

Earlier quoted context omitted.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

> Here's what the bare metal server didn't come with: [bunch of stuff I don't need] Exactly. Imagine paying for all that when all you need is bare metal. Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do. Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy. Buy what you ne…

best practice

Ugh. I hate that phrase. The translation into plain English is almost always "What I read in some blog" or "Because I want to" or "It's what our sales rep told us." Even from C-levels who should know better.

Re: Reclaiming the lost art of Linux server administration

#299

Post by sciurus »

milesvp wrote: post #176
viraptor wrote: post #165

Earlier quoted context omitted.

> when you pay for a hot standby you can't use it as a read only replica (true for AWS I'm not sure what you mean here. At least for MySQL you can have an instance configured as replica + read-only and used for reads. Aurora makes that automatic / transparent too with a separate read endpoint.

A hot standby is not a read replica. It's a set of servers running in a different Availability zone mirroring current prod, that is configured to automatically fail over to if the primary is offline. It's been a few years since I personally set this up in AWS, but at the time, those servers were completely unavailable to me, and basically doubled the cost of my production servers. The fact that a hot standby is usual…

They recently announced readable standby instances.

https://aws.amazon.com/blogs/database/readable-standby-insta...

Re: Reclaiming the lost art of Linux server administration

#300

Post by d0gsg0w00f »

readingnews wrote: post #75
bblb wrote: post #41

Earlier quoted context omitted.

I work in IT Operations of a big IT house. 100% local gov customers. We fully manage around 5000 pet servers. ~30 sysadmins and some of us do architecture designing also. There's also a separate networking team of about 10 network specialists. Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. My hunch is that the 5000 onprem pet servers are not going away any…

>>Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g…

You should learn some AWS and use it as a trojan horse to get those jobs. As a former old school sysadmin I really took to it because it's so modular and feels like Unix's "collection of simple tools piped together". Plus, any old school admin is a treasure on any cloud team because the need to drop to shell is unavoidable. People think they don't need sysadmins on cloud teams but behind every good cloud team are a few good sysadmins.
Post reply on HN