Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

181–190 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#181

Post by tormeh »

mark242 wrote: post #100
marcosdumay wrote: post #32

Earlier quoted context omitted.

> Entirely eliminating "system administration" from my duties... ... and adding "cloud administration". What is it with people doing completely one-sided analysis even when they experiment the thing by themselves? Is cloud administration less time consuming than system administration? That's not my experience, so I'm quite interested on how it got so.

> Is cloud administration less time consuming than system administration? Infinitely, and if you look at it from a startup lens it only makes sense. One needs to point only at the recent log4j incident. This is obviously a gigantic black swan event, but even just ongoing security patching at the OS level can be a full-time gig. There is absolutely no substitution for being able to ship code to a platform that just ru…

> ongoing security patching at the OS level can be a full-time gig

That's just a cronjob. I know some people don't like doing it that way, but that's on them. I've seen this work for years in production with minimal trouble.

Re: Reclaiming the lost art of Linux server administration

#182

Post by Hackbraten »

mark242 wrote: post #114
jimmaswell wrote: post #28

From my experience I would never recommend giving up control of your servers to some third party. Weeks wasted waiting for useless support teams to get back to you on something you could have fixed in 10 minutes if you had root. Opaque configuration issues you can't debug without said useless support team. Needing permission and approval for every little thing on prod. If I was ever at a high level in a company I'd n…

> If I was ever at a high level in a company I'd never go farther than some AWS load balancing or whatever on cloud instances we still have root on. Your competitors would salivate at this statement, fyi. Speed is a competitive advantage. AWS is not "let's rent a big ball of EC2 servers and call it a day", and anyone who treats it like that is going to get eaten alive. If you have not looked at -- for example -- Dyna…

Our customer (300k+ employees) switched to AWS a couple of years ago and I simply hate it so much. The “predictable, scalable” service we’re using, MSK, is a pain to develop against. Log files are all over the place, thanks to microservices and because no one, myself included, has a clue on how to make them manageable again. AWS’s graphical user interface is a constantly changing mess. I hate clicking my way through a GUI just so I can download individual log files manually.

I wonder how you folks manage to work with AWS and not hate it.

Re: Reclaiming the lost art of Linux server administration

#183

Post by tormeh »

TacticalCoder wrote: post #74

A very weird thread that degenerated into: "PaaS vs self-hosted/self-owned hardware" . I'm pretty sure most people sysadmin'ing their Linux servers are actually doing it with rented dedicated servers. TFA btw specifically mentions: "don't manage physical hardware" . Big companies like Hetzner and OVH have hundreds of thousands of servers and they're not the only players in that space. They don't take care of "everyth…

> it doesn't imply either "using zero cloud services"

Enter big-three cloud egress pricing. Designed to make sure that you have to go all-in.

Re: Reclaiming the lost art of Linux server administration

#184

Post by christophilus »

StillBored wrote: post #85
deepspace wrote: post #56

Earlier quoted context omitted.

> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has…

So, just buy another and leave it as a hot (or cold) standby in a different data-center. Or use AWS as the DR site an spin it up only if the local HW fails. This sounds expensive if your talking one server and vs a year of AWS charges, but is a tiny bump if it turns out you need to buy a dozen servers to replace a large AWS bill. Plus, I think most people underestimate how reliable server grade hardware is. Most of i…

> most people underestimate how reliable server grade hardware is

This. And there are millions of dollars of cloud marketing materials and programs that are at least partly to blame.

Re: Reclaiming the lost art of Linux server administration

#185

Post by NikolaeVarius »

0xCMP wrote: post #94
candiddevmike wrote: post #13

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

In my experience Pet servers are a good starting point (you really should _graduate_ from Pet servers into all the various immutable/cattle stuff), but it can quickly require discipline from the Admins. They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutab…

I find people dont know how amazing a "immutable" server fleet is until you've experienced it.

It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour.

It was amazing, never had to deal with something missing on a server or a config being wrong in a special case. Dozens of hosts just purring along with 0 downtime since the moment anything became unhealthy, hosts would start auto-booting and terminate the old instance.

Re: Reclaiming the lost art of Linux server administration

#186

Post by [deleted] »

VWWHFSfQ wrote: post #12

Regular SA and DBA jobs will be almost completely gone within a decade or so. Same as there are hardly any auto mechanics anymore because nobody can fix any of the new cars but the manufacturer. You'll only find those jobs at one of the handful of cloud companies. Nobody will know how to do anything for themselves anymore and all this experience and knowledge will be lost. There are no more actual administrators. Jus…

[deleted]

Re: Reclaiming the lost art of Linux server administration

#187

Post by bamboozled »

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

> I know where our data lives and no one can access it without a warrant and my explicit knowledge

As far as you know?

Your data is exposed to The Internet so someone could be accessing it.

Re: Reclaiming the lost art of Linux server administration

#188

Post by ozim »

senko wrote: post #160
viraptor wrote: post #139

Earlier quoted context omitted.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

> Here's what the bare metal server didn't come with: [bunch of stuff I don't need] Exactly. Imagine paying for all that when all you need is bare metal. Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do. Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy. Buy what you ne…

I am buying IaaS it is so nice to use VPS with snapshots every 4 hours that I don't have to think about.

I don't care where those snapshots are stored and how much space those take. In case I need to restore my IaaS provider gives me 2-click option to restore - one to click restore and 2nd to confirm. I sit and watch progress. I also don't care about hardware replacement and anything that connects to that. I have to do VPS OS updates but that is it.

I do my own data backups on different VPS of course just in case my provider has some issue, but from convenience perspective that IaaS solution is delivering more than I would ask for.

Re: Reclaiming the lost art of Linux server administration

#189

Post by bamboozled »

exdsq wrote: post #180
aledalgrande wrote: post #52

Earlier quoted context omitted.

I'm just curious, because to me it seems a little bit unrealistic. How do you handle traffic spikes, especially from the networking point of view? What kind of connection do you have? How do you make your service as fast for all customers around the world (saying you have a succesful Saas). How do you prevent a local blackout from taking down your service? Where do you store your backups, in case your building gets f…

In my first job as a system tech for an IT company in 2014 we had a backup process run at 17:30 and whichever admin left last would take the backup HDD with them home lol. It worked! There was also onsite redundancy with replicated windows servers in an office across the street, which was enough. Simpler times even just 8 years ago!

Which is ok if there isn’t a local disaster which wipes out your office and your friends?

Re: Reclaiming the lost art of Linux server administration

#190

Post by rovr138 »

ozim wrote: post #188
senko wrote: post #160

Earlier quoted context omitted.

> Here's what the bare metal server didn't come with: [bunch of stuff I don't need] Exactly. Imagine paying for all that when all you need is bare metal. Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do. Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy. Buy what you ne…

I am buying IaaS it is so nice to use VPS with snapshots every 4 hours that I don't have to think about. I don't care where those snapshots are stored and how much space those take. In case I need to restore my IaaS provider gives me 2-click option to restore - one to click restore and 2nd to confirm. I sit and watch progress. I also don't care about hardware replacement and anything that connects to that. I have to…

> Buy what you need
Post reply on HN