Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

251–260 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#251

Post by andrewmcwatters »

solatic wrote: post #102

Why can't people just pick the right tool for the job? The truth behind these managed services is that, for the correct usecases, they are VERY cheap. And for the wrong usecases, they are RIDICULOUSLY expensive. Most businesses have nightly cronjobs generating some kind of report that is then emailed to stakeholders. Why on Earth would you run a dedicated Linux box for that anymore? Glue a nightly trigger to AWS Lamb…

> Pick the right tool for the job.

This is such a tired expression. It basically means nothing in the industry, and exactly because of comments like yours.

Exactly who are you to say what my infrastructure desires are? Software is personal and people ignore this completely.

Re: Reclaiming the lost art of Linux server administration

#252

Post by NikolaeVarius »

Hackbraten wrote: post #182
mark242 wrote: post #114

Earlier quoted context omitted.

> If I was ever at a high level in a company I'd never go farther than some AWS load balancing or whatever on cloud instances we still have root on. Your competitors would salivate at this statement, fyi. Speed is a competitive advantage. AWS is not "let's rent a big ball of EC2 servers and call it a day", and anyone who treats it like that is going to get eaten alive. If you have not looked at -- for example -- Dyna…

Our customer (300k+ employees) switched to AWS a couple of years ago and I simply hate it so much. The “predictable, scalable” service we’re using, MSK, is a pain to develop against. Log files are all over the place, thanks to microservices and because no one, myself included, has a clue on how to make them manageable again. AWS’s graphical user interface is a constantly changing mess. I hate clicking my way through…

This is just a bad setup.

Ship logs others places, clicking in the GUI outside of testing things out is a mistake, use Terraform or CDK or Cloudformation.

I've managed very large fleets of servers and its dead simple if you have a good setup.

Re: Reclaiming the lost art of Linux server administration

#253

Post by ozim »

DarylZero wrote: post #198
ozim wrote: post #188

Earlier quoted context omitted.

I am buying IaaS it is so nice to use VPS with snapshots every 4 hours that I don't have to think about. I don't care where those snapshots are stored and how much space those take. In case I need to restore my IaaS provider gives me 2-click option to restore - one to click restore and 2nd to confirm. I sit and watch progress. I also don't care about hardware replacement and anything that connects to that. I have to…

Snapshots every 4 hours? That doesn't sound impressive at all. In 2022 that's laptop tier capability.

Well my dev laptop could run 1000x more traffic than what I need on my servers.

But of course I am running build tools and bunch of other dev tools locally that are not even installed on our servers.

Re: Reclaiming the lost art of Linux server administration

#254

Post by NikolaeVarius »

0xbadcafebee wrote: post #203

I still do not understand how anyone can become a software engineer and not stop to learn how an operating system or network works. I would have gone crazy if I'd never learned how network protocols work, or how web servers work, or virtual memory, or schedulers, security models, etc. It's like manufacturing tires without knowing how an engine works. Don't you want to know how torque and horsepower affect acceleratio…

Devs dont want to know. In the spirit of "DevOps" we needed to give Devs full admin access to Production servers. I have seen Staff Engineers do the most mind boggling things on these servers that cause issues.

Re: Reclaiming the lost art of Linux server administration

#255

Post by gerdesj »

"I picked nginx as my default webserver, although I hear Apache is also good."

In my opinion: when you have choice, get to know all the options (within reason). I have Apache as my default, purely because nginx didn't exist for many years. When nginx turned up, I gave it a while to calm down and now I deploy it quite often. I deploy something like 75% Apache and 25% nginx.

I tend to Apache from inertia but I quite like the clean easy setup for a simplish site with nginx - this is with Debian/Ubuntu style defaults, which do not favour nginx.

Re: Reclaiming the lost art of Linux server administration

#256

Post by zerocrates »

c2h5oh wrote: post #147
viraptor wrote: post #139

Earlier quoted context omitted.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

Of course there are a lot of benefits of using hosted databases. I like hosted databases and use them for both work and personal projects. What I have a problem with is: - the premium over bare metal is just silly - maximum vertical scaling being a rather small fraction of what you could get with bare metal - when you pay for a hot standby you can't use it as a read only replica (true for AWS and GCP, idk about Azure…

Readable standby instances is something AWS just recently added actually: https://aws.amazon.com/blogs/database/readable-standby-insta...

Though it seems to require you to have 3 instances rather than just letting you read from the standby... I don't quite get the rationale for that.

Re: Reclaiming the lost art of Linux server administration

#257

Post by justinclift »

Andrew_nenakhov wrote: post #124
bmurphy1976 wrote: post #110

Earlier quoted context omitted.

You don't need a 42u rack. You can run a cluster of Raspberry Pi's hidden in your basement ceiling rafters like me.

Racks are not that expensive, and are a very good way to keep things connected, powered, accessible and tidy. Heaps of Pis in rafters will quickly turn into a cable spaghetti hell tied into ugly knots.

Sounds like a new category of specialist rack is needed.

Introducing the... "RafterRack™" ;)

Re: Reclaiming the lost art of Linux server administration

#258

Post by [deleted] »

readingnews wrote: post #75
bblb wrote: post #41

Earlier quoted context omitted.

I work in IT Operations of a big IT house. 100% local gov customers. We fully manage around 5000 pet servers. ~30 sysadmins and some of us do architecture designing also. There's also a separate networking team of about 10 network specialists. Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. My hunch is that the 5000 onprem pet servers are not going away any…

>>Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g…

[deleted]

Re: Reclaiming the lost art of Linux server administration

#259

Post by BirAdam »

nonameiguess wrote: post #221
candiddevmike wrote: post #13

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

This works fine from the perspective of the admin, but think from the perspective of a business owner hiring admins. What happens when you have employee turnover? You don't want all of the knowledge needed to deploy your critical IT infrastructure stored in one guy's head or personal notebook. Maybe your admins were all disciplined enough to make sure they never made a single change without documenting it and kept ex…

Treat your people better and you may not lose them all.

Re: Reclaiming the lost art of Linux server administration

#260

Post by section_me »

belter wrote: post #157
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Not a fair comparison and you know it. Now add to the $200 month bare metal server, the yearly salary of the 3 admins you need to manage it. One as backup, one for day time, one for the night plus a weekend rotation. Add to the admin salaries social security, insurance and a margin of safety if one is unavailable due to sickness.

I think this is valid in many cases. A decent sysadmin in the US is (speculating) 200K USD, a good one in Europe is 120K+ euro. You have cheaper but they probably can't explaint a UDP datagram or know how ECMP over BGP works. It's the understanding of all of the underling tech which make them worth there weight in gold.
Post reply on HN