Reclaiming the lost art of Linux server administration • Hacker News Board

Re: Reclaiming the lost art of Linux server administration

#81

Post by unknown2374 » Fri, Jan 28, 2022, 8:03 PM UTC

aledalgrande wrote: post #60

unknown2374 wrote: post #53

Earlier quoted context omitted.

I find it very similar to how understanding of OS and hardware fundamentals can make one a much better software engineer, how infrastructure in the cloud/servers are setup helps make better design decisions. At least in my experience, my hobby of maintaining my own home server helped out immensely in my path in the industry due to knowing what tools are available when working on multi-faceted software designs.

It does, but you don't wanna have to deal with it constantly, if you want to be working on a lot of feature work as an application developer.

Definitely agree with you on that. Making use of layers of abstraction and delegation is absolutely necessary when working on more and more impactful work.

Re: Reclaiming the lost art of Linux server administration

#82

Post by jenscow » Fri, Jan 28, 2022, 8:03 PM UTC

encryptluks2 wrote: post #42

chasil wrote: post #3

-"As for scripting, commit to getting good at Bash." That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. This does not implement much of bash and will fail spectacularly on the simplest of scripts. This is also an issue on systems using Busybox. A useful approach to scripting is to grasp the POSIX shell first, then facets of bash and Korn as they are needed. -"As a practi…

> -"As a practical goal, you should be able to recreate your host with a single Bash script." I disagree with this. A single bash script configuring an entire hosts can be overly complex and very difficult to follow. As someone who has created complex bash scripts, this will become very time consuming and prevent you from making many changes without significant efforts. I'd suggest familiarizing yourself with tools l…

My take on that, was the host creation should be simple enough to only require bash.

Re: Reclaiming the lost art of Linux server administration

#83

Post by lamontcg » Fri, Jan 28, 2022, 8:04 PM UTC

netr0ute wrote: post #50

cure wrote: post #17

Earlier quoted context omitted.

Because you're not going to have a great time with /bin/sh (i.e. dash or the like) if you want to do anything more than very, very basic scripts.

Relying on bash is a recipe for non-portability.

If you're not publishing your scripts and you're running your own infrastructure, you probably don't care about portability at all.

Re: Reclaiming the lost art of Linux server administration

#84

Post by Karrot_Kream » Fri, Jan 28, 2022, 8:05 PM UTC

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I'm curious how much you're paying for business fiber. Do you have IP transit in your home, or is it just a business connection?

Re: Reclaiming the lost art of Linux server administration

#85

Post by StillBored » Fri, Jan 28, 2022, 8:07 PM UTC

deepspace wrote: post #56

warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has…

So, just buy another and leave it as a hot (or cold) standby in a different data-center. Or use AWS as the DR site an spin it up only if the local HW fails.

This sounds expensive if your talking one server and vs a year of AWS charges, but is a tiny bump if it turns out you need to buy a dozen servers to replace a large AWS bill.

Plus, I think most people underestimate how reliable server grade hardware is. Most of it gets retired because its functionally obsolete, not because a power supply/whatever fails. Which brings up the point, that the vast number of failures with server grade hardware are on replaceable components like power supplies, disks, SFP's, etc. Three or four years out those parts are available on the secondary markets frequently for pocket change.

Re: Reclaiming the lost art of Linux server administration

#86

Post by Yuioup » Fri, Jan 28, 2022, 8:08 PM UTC

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

Yes but you have talent and a lifetime of experience, plus space for a noisy 42u rack full of servers, but not everybody does...

Re: Reclaiming the lost art of Linux server administration

#87

Post by c2h5oh » Fri, Jan 28, 2022, 8:10 PM UTC

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month.

IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Re: Reclaiming the lost art of Linux server administration

#88

Post by guilhas » Fri, Jan 28, 2022, 8:11 PM UTC

I do run my home server, but it is definitely an investment, and for some use cases it might not be worth it

The hardware is the cheapest part, then you have to pay electricity, manage backups, fix raid problems, have a good internet. Pay attention to how the server is doing. And if you're serving a business, you have to be available debug any issue. Investing a lot of time you could be actually working on the project

But definitely most devs should have a small home server for trying unimportant things. Nothing complicated, just keep the standard hardware config. There are second hand servers available for 50$. Install some Linux and have it running 24/7. Quite fun experimenting and hosting simple things

Re: Reclaiming the lost art of Linux server administration

#89

Post by acoard » Fri, Jan 28, 2022, 8:16 PM UTC

rank0 wrote: post #44

warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

Do you have a static IP? I have a homelab too but getting “enterprise grade” service from comcast seems to be my biggest barrier to scaling without leaning on aws.

Dynamic DNS (dDNS) works here[0]. You have free services like no-ip, and also most paid domain registrars support this. I know both Namecheap and AWS Route 53 support it if you want it at your own domain. Essentially, it's a cron curling with an API key from the host machine, that's it. Works great in my experience.

Re: Reclaiming the lost art of Linux server administration

#90

Post by Nextgrid » Fri, Jan 28, 2022, 8:18 PM UTC

trabant00 wrote: post #19

VWWHFSfQ wrote: post #12

Regular SA and DBA jobs will be almost completely gone within a decade or so. Same as there are hardly any auto mechanics anymore because nobody can fix any of the new cars but the manufacturer. You'll only find those jobs at one of the handful of cloud companies. Nobody will know how to do anything for themselves anymore and all this experience and knowledge will be lost. There are no more actual administrators. Jus…

I've been hearing this for the past 20 years. And now my sysadmin skills are more and more in demand. For the past 5 years or so I started making more money than a dev because of supply and demand. Rent to AWS actually drives demand up quite a lot since the bills are huge and very few people understand what is under the hood and how it can be optimized. I doubt very much things will change in the near future. In the…

Out of curiosity, how do you find old-school sysadmin gigs? I find that everything nowadays requires knowing the specifics of a particular cloud and their managed services as opposed to raw Linux or networking knowledge.