Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

241–250 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#241

Post by freedomben »

wwweston wrote: post #156
StillBored wrote: post #85

Earlier quoted context omitted.

So, just buy another and leave it as a hot (or cold) standby in a different data-center. Or use AWS as the DR site an spin it up only if the local HW fails. This sounds expensive if your talking one server and vs a year of AWS charges, but is a tiny bump if it turns out you need to buy a dozen servers to replace a large AWS bill. Plus, I think most people underestimate how reliable server grade hardware is. Most of i…

> Or use AWS as the DR site an spin it up only if the local HW fails. Yep. This seems like the obvious setup to me: 1) make the usual case as economical as possible (and ownership and the associated control will probably help here, unless you have to lease the expertise too) 2) outsource the exceptional case (ownership is less likely to matter here, and will matter for less time even if it does)

This is exactly what many of the really big enterprises do. At their scale the cloud becomes laughable. But as a backup, it's the clear way to go. Primary on prem with automatic failover to the cloud.

I worked with a large customer to help build this for them. OpenShift running on-prem, but they had some failover equipment that would ansible them an OpenShift cluster on aws. Depending on the nature of the failure it did take a little time to fail over, but 15 to 30 mins of downtime in the event of a catastrophic failure is often worth it to save hundreds of millions per year.

Re: Reclaiming the lost art of Linux server administration

#242

Post by lambdaba »

trabant00 wrote: post #19
VWWHFSfQ wrote: post #12

Regular SA and DBA jobs will be almost completely gone within a decade or so. Same as there are hardly any auto mechanics anymore because nobody can fix any of the new cars but the manufacturer. You'll only find those jobs at one of the handful of cloud companies. Nobody will know how to do anything for themselves anymore and all this experience and knowledge will be lost. There are no more actual administrators. Jus…

I've been hearing this for the past 20 years. And now my sysadmin skills are more and more in demand. For the past 5 years or so I started making more money than a dev because of supply and demand. Rent to AWS actually drives demand up quite a lot since the bills are huge and very few people understand what is under the hood and how it can be optimized. I doubt very much things will change in the near future. In the…

Hey I'm super interested in such sysadmin work if you could describe it here or share it on another channel (email in bio). Many thanks.

Re: Reclaiming the lost art of Linux server administration

#243

Post by freedomben »

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

You can do an amazing amount of work on a raspberry pi these days. I have a couple of R620s that were running several services. I lost a drive and one went down so I stood up a Pi4 (8GB) to quickly get things back up, and I was amazed that it handled a huge piece of the load, much more than I expected. Definitely worth a try for somebody just starting out.

Re: Reclaiming the lost art of Linux server administration

#244

Post by bblb »

LoveGracePeace wrote: post #143
bblb wrote: post #77

Earlier quoted context omitted.

Caddyserver is Apache/Nginx killer and we will talk about Caddy in a couple of years as if it's been always the default ("why did we kept fighting with apache/nginx all those years, silly us"). Seriously. It's just a completely different way to think about web servers and automation. I'm just amazed it took all these years to emerge.

I love Apache, do not love Nginx and won't be looking at Caddy (it's written in Golang). Apache (even Nginx) are easy to set up reverse proxy and can simultaneously serve static content and serve as the https certificate main, as well as a few dozen other things like load balancing and rate limiting, etc.

I'm just a "user" aka sysadmin who sets up stuff, so can't really comment on the Golang. Even just the fully automated LE cert magic in Caddy is a win in average on-premises environment, where TLS certs are still a big pain and massive manual work drain. And the JSON API automation is a big plus also. At least for our use cases Caddy resolves one of the biggest issues.

Re: Reclaiming the lost art of Linux server administration

#245

Post by icedchai »

belter wrote: post #157
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Not a fair comparison and you know it. Now add to the $200 month bare metal server, the yearly salary of the 3 admins you need to manage it. One as backup, one for day time, one for the night plus a weekend rotation. Add to the admin salaries social security, insurance and a margin of safety if one is unavailable due to sickness.

Everything but the most mission critical applications can get away without any of that. I get your point though.

Re: Reclaiming the lost art of Linux server administration

#246

Post by oblio »

daveevad wrote: post #226
mark242 wrote: post #100

Earlier quoted context omitted.

> Is cloud administration less time consuming than system administration? Infinitely, and if you look at it from a startup lens it only makes sense. One needs to point only at the recent log4j incident. This is obviously a gigantic black swan event, but even just ongoing security patching at the OS level can be a full-time gig. There is absolutely no substitution for being able to ship code to a platform that just ru…

> in the future, 100% of the code that you write will be business logic How much business logic is there across all businesses? I'm aware of a joke at Google about how it could reduce 90% of the employees without any impact on the business.

> How much business logic is there across all businesses?

Immensely more than tech logic, that's for sure.

Just implementing and updating something like a single country's tax code produces more lines of code than the entire Linux kernel, that devs like to brag about :-)

Re: Reclaiming the lost art of Linux server administration

#247

Post by BirAdam »

viraptor wrote: post #139
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

Everything has a cost. The cost of the cloud is that you’re out of control, the cloud provider owns your stuff, and a single vulnerability can make half the Earth get pwned. Whoops.

EDIT: I do use cloud services for some stuff. My point isn’t being anti-cloud, just that nothing is perfect.

Re: Reclaiming the lost art of Linux server administration

#248

Post by oblio »

chasil wrote: post #230
notyourday wrote: post #178

Earlier quoted context omitted.

> Android doesn't allow GPL code in userland, and the installed base is massive. You aren't administering Android devices. Stop obsessing about writing portable scripts. Write scripts for the targets that you are going to run them on.

I run Lineage, and use a number of scripts. Stop ignoring standards. They exist for important reasons. Invent your existentialism in some other realm.

> Stop ignoring standards.

That ship sailed when they froze that standard in time circa around 2008.

People will always choose convenience over correctness. Fighting against that is fighting against human nature and it's a battle lost before it's begun.

Re: Reclaiming the lost art of Linux server administration

#249

Post by hosteur »

tormeh wrote: post #183
TacticalCoder wrote: post #74

A very weird thread that degenerated into: "PaaS vs self-hosted/self-owned hardware" . I'm pretty sure most people sysadmin'ing their Linux servers are actually doing it with rented dedicated servers. TFA btw specifically mentions: "don't manage physical hardware" . Big companies like Hetzner and OVH have hundreds of thousands of servers and they're not the only players in that space. They don't take care of "everyth…

> it doesn't imply either "using zero cloud services" Enter big-three cloud egress pricing. Designed to make sure that you have to go all-in.

And a good reason to stay out.

Re: Reclaiming the lost art of Linux server administration

#250

Post by chasil »

ericbarrett wrote: post #232
chasil wrote: post #73

Earlier quoted context omitted.

If you want a portable script, then you don't want that behavior.

Then I wouldn't put #!/bin/bash at the top.

Your problem there is Ubuntu/Debian.

A bash script that runs as /bin/sh might not run as /bin/bash, until you set POSIXLY_CORRECT.

Easiest solution there is to move it to a better shell, usually a Korn variant.

Post reply on HN