Reclaiming the lost art of Linux server administration • Hacker News Board

Re: Reclaiming the lost art of Linux server administration

#161

Post by dane-pgp » Fri, Jan 28, 2022, 9:44 PM UTC

thepra wrote: post #121

I would argue to you don't even need to put that much effort into learning bash scripting, you can totally get away with knowing systemd, journalctl, nginx, apt, ssh and docker and how to run them through bash. Everything else is per-software files configuration and running commands from the software setup documentation. Plus, I would run a server with a DE simply because I want to be able to look into databases with…

> knowing systemd, journalctl, nginx, apt, ssh and docker and how to run them through bash.

Or, the way things are going, systemd, systemd[1], systemd[2], systemd[3], systemd[4] and systemd[5].

[1] https://www.freedesktop.org/software/systemd/man/journalctl.... [2] https://www.freedesktop.org/software/systemd/man/systemd-jou... [3] https://www.freedesktop.org/software/systemd/man/systemd-mac... [4] https://www.freedesktop.org/software/systemd/man/systemd-log... [5] https://www.freedesktop.org/software/systemd/man/systemd-nsp...

Re: Reclaiming the lost art of Linux server administration

#162

Post by 0xCMP » Fri, Jan 28, 2022, 9:44 PM UTC

candiddevmike wrote: post #120

0xCMP wrote: post #94

Earlier quoted context omitted.

In my experience Pet servers are a good starting point (you really should _graduate_ from Pet servers into all the various immutable/cattle stuff), but it can quickly require discipline from the Admins. They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutab…

Sounds like you need a new CTO.

As turns out a lead developer can't unilaterally change the CTO. Not sure how it works for you. I can control tech, direction, etc. or move on to another job.

I chose to work with the CTO/Team to figure out a solution everyone could live with. I even chose a more annoying solution (Packer) initially just to make sure people felt comfortable and avoid changing things anymore than I had to.

Re: Reclaiming the lost art of Linux server administration

#163

Post by [deleted] » Fri, Jan 28, 2022, 9:44 PM UTC

Andrew_nenakhov wrote: post #124

bmurphy1976 wrote: post #110

Earlier quoted context omitted.

You don't need a 42u rack. You can run a cluster of Raspberry Pi's hidden in your basement ceiling rafters like me.

Racks are not that expensive, and are a very good way to keep things connected, powered, accessible and tidy. Heaps of Pis in rafters will quickly turn into a cable spaghetti hell tied into ugly knots.

[deleted]

Re: Reclaiming the lost art of Linux server administration

#164

Post by jjav » Fri, Jan 28, 2022, 9:51 PM UTC

jimmaswell wrote: post #28

From my experience I would never recommend giving up control of your servers to some third party. Weeks wasted waiting for useless support teams to get back to you on something you could have fixed in 10 minutes if you had root. Opaque configuration issues you can't debug without said useless support team. Needing permission and approval for every little thing on prod. If I was ever at a high level in a company I'd n…

Yes, the absolute opaqueness of so-called serverless is a huge hit to productivity.

Numerous times there's something weird going on and you're stuck trying to guess and retry based on largely useless logs until it somehow works better but you never really know what the root cause truly was.

Meanwhile on my own server I'll ssh in and have complete visibility, I can trace and dump network traffic, bpftrace userspace and kernel code, attach debuggers, there's no limit to visibility.

Yes lambda/serverless saves you a day or three in initial setup but you'll pay that time back with 10-100x interest as soon as you need to debug anything.

Re: Reclaiming the lost art of Linux server administration

#165

Post by viraptor » Fri, Jan 28, 2022, 9:52 PM UTC

c2h5oh wrote: post #147

viraptor wrote: post #139

Earlier quoted context omitted.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

Of course there are a lot of benefits of using hosted databases. I like hosted databases and use them for both work and personal projects. What I have a problem with is: - the premium over bare metal is just silly - maximum vertical scaling being a rather small fraction of what you could get with bare metal - when you pay for a hot standby you can't use it as a read only replica (true for AWS and GCP, idk about Azure…

> when you pay for a hot standby you can't use it as a read only replica (true for AWS

I'm not sure what you mean here. At least for MySQL you can have an instance configured as replica + read-only and used for reads. Aurora makes that automatic / transparent too with a separate read endpoint.

Re: Reclaiming the lost art of Linux server administration

#166

Post by senko » Fri, Jan 28, 2022, 9:52 PM UTC

deepspace wrote: post #56

warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has…

> And when some component of the server fails, your app is unavailable until you can repair it.

So you have some downtime. Big deal. If this happens once every few years and you need a day to repair it, your uptime is still better than AWS.

Not just everyone hosts a realtime API millions of users depend on every second of the day.

Re: Reclaiming the lost art of Linux server administration

#167

Post by cerved » Fri, Jan 28, 2022, 9:52 PM UTC

sseagull wrote: post #93

Yuioup wrote: post #86

Earlier quoted context omitted.

Yes but you have talent and a lifetime of experience, plus space for a noisy 42u rack full of servers, but not everybody does...

People who have been woodworking for decades can own very expensive tools so that they can create very complicated things. People who are experts in cars can own very expensive cars and tools to tune them. People who have been working in music can have very expensive instruments and expensive headphones, microphones, sequencers, etc. We seem to be looking down on experienced "computer experts" and wanting to take the…

bruh nobody's looking down on people running they're own metal

Server hardware is fun but it's not trivial to manage, buy or run.

So when someone talks about how they've managed servers for 2 decades, own a house where they can install a 42 rack and how much better it is than a hosted solution. A lot of people rightly point out that this is hardly feasible for most people

Re: Reclaiming the lost art of Linux server administration

#168

Post by aparks517 » Fri, Jan 28, 2022, 9:53 PM UTC

belter wrote: post #157

c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Not a fair comparison and you know it. Now add to the $200 month bare metal server, the yearly salary of the 3 admins you need to manage it. One as backup, one for day time, one for the night plus a weekend rotation. Add to the admin salaries social security, insurance and a margin of safety if one is unavailable due to sickness.

Around here, you'd spread the work of one new server to folks you've already got or spread the cost of new hires to a large number of servers.

I think it's also worth considering that many outfits wouldn't get good value from the 24/365 coverage you propose and don't care to pay for it.

Re: Reclaiming the lost art of Linux server administration

#169

Post by zepearl » Fri, Jan 28, 2022, 9:53 PM UTC

I don't think that the core of the article is about pros&cons of the managed/unmanaged/virtualized/dedicated server/service approach, but about "why it would be a good idea to have your own dedicated or virtualized server (at least for a while), which is to assimilate know-how" (which can then be used in more abstract setups).

The total flexibility of such a server (compared to un/managed services) is a (great) bonus (not only at the beginning).

Re: Reclaiming the lost art of Linux server administration

#170

Post by reacharavindh » Fri, Jan 28, 2022, 9:54 PM UTC

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

You can also pick and choose how far you go with self management. To me, as much fun as it is, I can’t be bothered to run the physical server in my home(not because I can’t, but because I won’t). That’s why my solution is to lease a bare metal server on Hetzner, pay EUR 38/month and host all my personal projects on it. This way I get a reliable network and operation for my server and I still hold reasonable control over my data and what I run.