Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

211–220 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#211

Post by [deleted] »

Yuioup wrote: post #86
lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

Yes but you have talent and a lifetime of experience, plus space for a noisy 42u rack full of servers, but not everybody does...

[deleted]

Re: Reclaiming the lost art of Linux server administration

#212

Post by deckard1 »

exdsq wrote: post #175
warent wrote: post #131

Earlier quoted context omitted.

This is an important point. My customers don't love outages (who does?) but I've had them and it doesn't really hurt that badly. My products aren't that critical. They're understanding as long as you communicate.

Plus they still happen on AWS (or other critical bits like GitHub) so you’re not immune anyway

github took down our eng dept probably half a dozen times last year. I don't even see how they hit 3 nines last year. It was insane. Even before that we had to switch off Azure. That was a sad joke of a service. They must be running their entire ops out of a log cabin connected with the bare minimum of cabling and bandwidth necessary.

That's also the thing the serverless/API-all-the-things/cloud promoters don't get. This interconnected web of services is incredibly fragile. It's to the point that every day is a new failure. One day github is down. Next day your CI process breaks because Docker shits the bed. Next day your E2E service is hosed. Following day you hit an API limit and need to go dump more money into the firepit. Everything is broken all the time.

Re: Reclaiming the lost art of Linux server administration

#213

Post by sokoloff »

orf wrote: post #55
qmarchi wrote: post #47

Earlier quoted context omitted.

Hey, you actually have a few options, notably, doing nothing! Comcast doesn't actually change your public IP address between DHCP renewals and thus it's effectively static. The only time that it'll change is when the modem is powered off for an amount of time, or the upstream DOCSIS concentrator is powered off for maintenance or otherwise.

So: arbitrarily and without warning.

Sure; however, I’ve had the same IP on Comcast residential for a little over a decade. There are many endeavors that could fade that low a frequency of IP changing.

Re: Reclaiming the lost art of Linux server administration

#214

Post by dilyevsky »

c2h5oh wrote: post #137
papito wrote: post #111

Earlier quoted context omitted.

LOL. Priceless. Having these skills is very valuable. Us old farts used to do a lot with what today would be called "bootstrapped". Scarcity is no longer a "problem", except that it is. Scarcity keeps you sharp, efficient, on the edge - where you need to be. It's also - cheaper.

Who would have guessed having local, low latency, high iops drives would be better than VM using iSCSI-attached drive for storage, right? ;-)

Tbf iscsi and especially it’s future demise - nvmeof can be made quite fast. The problem that in gp2/3, gce-pd cases it’s probably not some native storage but another layer of software-based distributed store which makes stuff slow/expensive

Re: Reclaiming the lost art of Linux server administration

#215

Post by Underphil »

bcrosby95 wrote: post #133
StillBored wrote: post #85

Earlier quoted context omitted.

So, just buy another and leave it as a hot (or cold) standby in a different data-center. Or use AWS as the DR site an spin it up only if the local HW fails. This sounds expensive if your talking one server and vs a year of AWS charges, but is a tiny bump if it turns out you need to buy a dozen servers to replace a large AWS bill. Plus, I think most people underestimate how reliable server grade hardware is. Most of i…

Yeah. We run servers into the ground where I work. We have around 20 of them. Average age is around 11 years old. Oldest is around 18.

Holy crap. I work in a relatively young sector and we retire them no later than 5 years. Do you mind sharing the type of business you work for?

Re: Reclaiming the lost art of Linux server administration

#216

Post by dgb23 »

belter wrote: post #157
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Not a fair comparison and you know it. Now add to the $200 month bare metal server, the yearly salary of the 3 admins you need to manage it. One as backup, one for day time, one for the night plus a weekend rotation. Add to the admin salaries social security, insurance and a margin of safety if one is unavailable due to sickness.

Cloud services don't need management or specialized workers?

Re: Reclaiming the lost art of Linux server administration

#217

Post by ReactiveJelly »

viraptor wrote: post #139
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

Some of these strike me as things where the software _could_ exist, but it's either not FOSS, or I haven't heard of it yet.

For instance, there are LVM2 snapshots. Maybe those do affect performance. If the cost difference is big enough though, couldn't you just account for that in the budget?

I agree that literal "bare metal" sucks, but self-hosted with cloud characteristics (containers, virtualization) is not totally obsolete.

Re: Reclaiming the lost art of Linux server administration

#218

Post by gerdesj »

DarylZero wrote: post #198
ozim wrote: post #188

Earlier quoted context omitted.

I am buying IaaS it is so nice to use VPS with snapshots every 4 hours that I don't have to think about. I don't care where those snapshots are stored and how much space those take. In case I need to restore my IaaS provider gives me 2-click option to restore - one to click restore and 2nd to confirm. I sit and watch progress. I also don't care about hardware replacement and anything that connects to that. I have to…

Snapshots every 4 hours? That doesn't sound impressive at all. In 2022 that's laptop tier capability.

What on earth are you running on your laptop?

Re: Reclaiming the lost art of Linux server administration

#219

Post by Underphil »

0xCMP wrote: post #162
candiddevmike wrote: post #120

Earlier quoted context omitted.

Sounds like you need a new CTO.

As turns out a lead developer can't unilaterally change the CTO. Not sure how it works for you. I can control tech, direction, etc. or move on to another job. I chose to work with the CTO/Team to figure out a solution everyone could live with. I even chose a more annoying solution (Packer) initially just to make sure people felt comfortable and avoid changing things anymore than I had to.

I have to give you kudos in being more polite and reserved in your reply than I would ever be able to.

Re: Reclaiming the lost art of Linux server administration

#220

Post by Underphil »

madjam002 wrote: post #95

Honestly after discovering NixOS I have a new found joy of administering Linux servers. It's easy and painless, everything is declarative and versioned, and new machines can be set up for new projects or scaling in a matter of minutes. This "cattle not pets" mentality doesn't make sense for everything and is highly inefficient if the OS itself seamlessly supports immutable workloads and configuration.

As someone who has to deal with legacy services from time to time I couldn't agree more. In certain industries, legacy software just doesn't scale at all.
Post reply on HN