Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

361–370 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#361

Post by kafkaIncarnate »

bamboozled wrote: post #189
exdsq wrote: post #180

Earlier quoted context omitted.

In my first job as a system tech for an IT company in 2014 we had a backup process run at 17:30 and whichever admin left last would take the backup HDD with them home lol. It worked! There was also onsite redundancy with replicated windows servers in an office across the street, which was enough. Simpler times even just 8 years ago!

Which is ok if there isn’t a local disaster which wipes out your office and your friends?

If there's a local disaster that wipes out your office and your friends and all you can think about is data then you should ask a professional to screen you for sociopathy and whether you have any treatable mental illness.

No shame in it, I myself have one, but you shouldn't be concerned with this scenario even if it happens.

Life is too short to waste on ruining your life over planning on how to ruin your life even further after a rare life ruining event has occurred. I doubt you'd be working, or having any real coherent thoughts, for many years in that scenario.

Re: Reclaiming the lost art of Linux server administration

#362

Post by mariushop »

When reporting to one of our clients for admining we fount that it was about 3 times cheaper to self-host their services (webapps, sites, email, fileserver, backup) than it was to migrate to cloud. Although they initially went this road because of compliance with some banking clients, they're pretty happy saving heaps on infra.

Re: Reclaiming the lost art of Linux server administration

#363

Post by kafkaIncarnate »

christophilus wrote: post #194
tormeh wrote: post #181

Earlier quoted context omitted.

> ongoing security patching at the OS level can be a full-time gig That's just a cronjob. I know some people don't like doing it that way, but that's on them. I've seen this work for years in production with minimal trouble.

It’s not even a cron job. It’s a single apt install.

log4j was different because each proprietary vendor has their own prepackaged version of it for some reason. Even SAN software or VMware.

Still, the mitigation was like 4 hours to hunt and disable, and 2 more hours when a full patch came around. Not too difficult.

It was only time consuming for those who worked in Cloud providers where all this crap is centralized and understaffed. In real world scenarios there were tons of teams completely unaffected.

Re: Reclaiming the lost art of Linux server administration

#364

Post by kafkaIncarnate »

ihateolives wrote: post #349
Ku_las33 wrote: post #304

Earlier quoted context omitted.

>> (it's written in Golang) That means no memory related security bugs, which is a huge plus.

Not really. JVM is similar in this regard and one of the thing we wrote in our basic Java course was the code that produces memory leak. And when you have memory leak you also may end up with other fun things like applications behaving unexpectedly in low memory conditions.

Java has a garbage collector, you shouldn't have memory leaks unless you are just incorrectly allocating memory, which isn't technically a leak which refers to allocated memory with no reference. Without a reference the garbage collector would pick it up.

Re: Reclaiming the lost art of Linux server administration

#365

Post by dilyevsky »

bushbaba wrote: post #347
dilyevsky wrote: post #214

Earlier quoted context omitted.

Tbf iscsi and especially it’s future demise - nvmeof can be made quite fast. The problem that in gp2/3, gce-pd cases it’s probably not some native storage but another layer of software-based distributed store which makes stuff slow/expensive

But also less prone to single drive failures. And easily auto expanding storage sizes. Everything is a trade off.

Except people end up running systems on them that have their own replication built in like kafka, cockroach etc. In all those cases you’d be better off with your node just hard crashing if disk goes bad rather than the latency penalty that inevitably follows the failover

Re: Reclaiming the lost art of Linux server administration

#366

Post by kafkaIncarnate »

MonaroVXR wrote: post #286
0xCMP wrote: post #94

Earlier quoted context omitted.

In my experience Pet servers are a good starting point (you really should _graduate_ from Pet servers into all the various immutable/cattle stuff), but it can quickly require discipline from the Admins. They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutab…

Do you have any tips in auto installing an OS on a server/desktop? I'm completely missing. I have searched arpund and I have some solutions, but back in my head, some people have something else. The part before Ansible or puppet to kick in.

1) Rocky Linux

2) Debian

Re: Reclaiming the lost art of Linux server administration

#367

Post by gscott »

b112 wrote: post #295
primax wrote: post #290

Earlier quoted context omitted.

Also physical security, power, physical rack cost, networking, salaries to manage everything, and all those other hidden costs

There are no "hidden costs". They're well known. Factor them all in, and "the cloud" is 1000s to millions of times more expensive than bare metal. I've seen people pay $100k/month, for services which could run on an $140/month bare metal server failover pair. Meanwhile, people still spend enormous resources managing "the cloud", writing code to deploy to the cloud, dealing with edge cases in the cloud. There are no s…

I was watching a video by Neil Patel and he pays 130k a month to Aws for one website, UberSuggest. Now that is a guy with money to burn. He could put that on bare metal and use the savings to buy a hotel and food for the homeless.

Re: Reclaiming the lost art of Linux server administration

#368

Post by kortilla »

viraptor wrote: post #139
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

Don’t forget all of the outages associated with all of that crap you don’t need.

Re: Reclaiming the lost art of Linux server administration

#369

Post by shirro »

We have returned to "You never get fired for buying IBM" but with various cloud services and layers of abstraction, profit taking, obfuscation and performance penalties. I am too old to care. I run my family servers to keep my skills and earn pocket money from time to time. I never learned to become a salesperson for Gsuite, Azure, AWS, Cloudflare Docker, whatever. Don't really care. I skip the majority of the blog spam articles posted to HN because many push the same nonsense. Massive industry deskilling and a race to the bottom. Steer clear kids and look for a good trade.

Re: Reclaiming the lost art of Linux server administration

#370

Post by PeterisP »

section_me wrote: post #260
belter wrote: post #157

Earlier quoted context omitted.

Not a fair comparison and you know it. Now add to the $200 month bare metal server, the yearly salary of the 3 admins you need to manage it. One as backup, one for day time, one for the night plus a weekend rotation. Add to the admin salaries social security, insurance and a margin of safety if one is unavailable due to sickness.

I think this is valid in many cases. A decent sysadmin in the US is (speculating) 200K USD, a good one in Europe is 120K+ euro. You have cheaper but they probably can't explaint a UDP datagram or know how ECMP over BGP works. It's the understanding of all of the underling tech which make them worth there weight in gold.

The sysadmin you need to manage your cloud infrastructure isn't going to be any cheaper, and if you want high-reliability 24/7, you still need your own staff on call 24/7, so you still need multiple sysadmins.

If some cloud setup can halve your sysadmin labor, then a company with 20 sysadmins can go to 10 sysadmins, but a company with 1, 2 or 3 sysadmins still needs all of them.

Post reply on HN