Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

301–310 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#301

Post by axiolite »

christophilus wrote: post #262
rad_gruchalski wrote: post #208

Earlier quoted context omitted.

I’ve picked up a second hand Dell R720 last year. Best purchase in years. It cost me €1500 for 48 cores and 192GB RAM.

I want one. What’s it done to your electricity bill, though?

If you want to use 4Kn drives, SSDs (w/trim), or easily switch between RAID and HBA modes, I would recomend an R730 with H730 controller instead. They're available on eBay with 128GB of RAM for just over $1,000 USD. If you don't want those features, an R720 is a good choice, and around half the price.

Idle power consumption of an R730 with no drives is typically about 85W, though higher-end CPUs drive that up a bit. The older R720s are a bit higher in the 100W range. When the extended warranties expire, high-spec 2nd hand equipment floods the market.

Re: Reclaiming the lost art of Linux server administration

#302

Post by rr808 »

akira2501 wrote: post #7
SkipperCat wrote: post #2

Time is money, and the more time I spend on infrastructure, the less time I spend on product. And thus is born the incredible demand of infrastructure as a service. Thankfully one person's cloud is another person's on prem infrastructure so sysadmin skills will always be in demand. From my perspective in enterprise computing, I now see people taking 2 paths. One where they become super deep sysadmins and work on infr…

That's when it clicked for me.. comparing my hourly salary rate vs. the cost of running these services "in the cloud." Entirely eliminating "system administration" from my duties was absolutely a net win for me and our team.

To me System administration is much simpler than cloud administration. Especially for security, I'm never confident cloud applications are fully secure.

Re: Reclaiming the lost art of Linux server administration

#303

Post by selfmodruntime »

section_me wrote: post #260
belter wrote: post #157

Earlier quoted context omitted.

Not a fair comparison and you know it. Now add to the $200 month bare metal server, the yearly salary of the 3 admins you need to manage it. One as backup, one for day time, one for the night plus a weekend rotation. Add to the admin salaries social security, insurance and a margin of safety if one is unavailable due to sickness.

I think this is valid in many cases. A decent sysadmin in the US is (speculating) 200K USD, a good one in Europe is 120K+ euro. You have cheaper but they probably can't explaint a UDP datagram or know how ECMP over BGP works. It's the understanding of all of the underling tech which make them worth there weight in gold.

No Sysadmin in Europe is being paid 120k Euro. It‘s learned trade and you‘d be lucky to pull 70k.

Re: Reclaiming the lost art of Linux server administration

#304

Post by Ku_las33 »

LoveGracePeace wrote: post #143
bblb wrote: post #77

Earlier quoted context omitted.

Caddyserver is Apache/Nginx killer and we will talk about Caddy in a couple of years as if it's been always the default ("why did we kept fighting with apache/nginx all those years, silly us"). Seriously. It's just a completely different way to think about web servers and automation. I'm just amazed it took all these years to emerge.

I love Apache, do not love Nginx and won't be looking at Caddy (it's written in Golang). Apache (even Nginx) are easy to set up reverse proxy and can simultaneously serve static content and serve as the https certificate main, as well as a few dozen other things like load balancing and rate limiting, etc.

>> (it's written in Golang)

That means no memory related security bugs, which is a huge plus.

Re: Reclaiming the lost art of Linux server administration

#305

Post by [deleted] »

Underphil wrote: post #223
pier25 wrote: post #136

I know how to setup a basic VPS with firewalls, nginx, etc, but I'm scared to death to do that for a production environment available online.

It's harder if you didn't learn back in the 90s or early 2000s. Getting hit by a bad actor was considerably less likely so we were able to learn from our mistakes without a significant cost. There's much less of a margin for error now.

[deleted]

Re: Reclaiming the lost art of Linux server administration

#306

Post by axiolite »

belter wrote: post #157
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Not a fair comparison and you know it. Now add to the $200 month bare metal server, the yearly salary of the 3 admins you need to manage it. One as backup, one for day time, one for the night plus a weekend rotation. Add to the admin salaries social security, insurance and a margin of safety if one is unavailable due to sickness.

> Now add to the $200 month bare metal server, the yearly salary of the 3 admins you need to manage it.

That's the line the clouds, MSPs, and service contract providers sell you on, but it's never true. When you need help you're getting some minimum wage slave who hired on 6 months ago, and is struggling to juggle the dozens of other clients competing with you for his time, too. When you don't need help, they get to take your money for doing nothing.

Anything your provider can do for you, you can do at less cost. Cut out the middle man and you're far better off.

Re: Reclaiming the lost art of Linux server administration

#307

Post by nirvdrum »

akira2501 wrote: post #7
SkipperCat wrote: post #2

Time is money, and the more time I spend on infrastructure, the less time I spend on product. And thus is born the incredible demand of infrastructure as a service. Thankfully one person's cloud is another person's on prem infrastructure so sysadmin skills will always be in demand. From my perspective in enterprise computing, I now see people taking 2 paths. One where they become super deep sysadmins and work on infr…

That's when it clicked for me.. comparing my hourly salary rate vs. the cost of running these services "in the cloud." Entirely eliminating "system administration" from my duties was absolutely a net win for me and our team.

I'm curious how you completely eliminated it. I've heard the claim from others, but it hasn't matched my experience at several companies over the past 13 years (the first place I worked at using AWS was in 2009).

While cloud may have a lot of advantages, I don't think it's trivial to run or manage. The AWS dashboard is simply overwhelming. Trying to decide between different, but overlapping services is time consuming. And while you can rebuild somewhat easily, you're also almost certainly going to have to do that as you learn about Amazon's little quirks. Your general RDBMS experience doesn't map to DynamoDB very well and you'll be in for a rough time when you learn you can't just add a new index or whatever.

Then you have all these provider-specific APIs. My experience with both Amazon and Google is that their services will return errors that they claim should be impossible, so you get to have fun debugging that in a service that you don't manage. Your application will invariably add a bunch of handling for exceptional cases and accumulate your best guess about how this impossible situation came about.

Then you have the constantly shifting devops orchestration tooling space and "best practices". I've lost track of the number of times I've needed to pull my Terraform state file and manually edit this gigantic JSON file because some plugin updated an internal struct in an incompatible way.

I'm sure there are people that get an environment up and running in one of IaaS platforms using just the web console. I've never seen it managed that way at any company I've been at. Instead, the devs own the IaaC stuff. It's certainly easier to be in multiple regions that way, but I have a hard time believing any time or money is really saved. Sure, no dedicated ops people, but now your expensive devs have to deal with and probably be on call. Moreover, all that archaic time-sucking Unix admin knowledge acquisition everyone is worried about is just replaced by time-sucking knowledge acquisition of a proprietary service and all its quirks.

Maybe we're talking about different levels of "cloud"? I can buy that Heroku is easier than AWS.

Re: Reclaiming the lost art of Linux server administration

#309

Post by justsomehnguy »

christophilus wrote: post #262
rad_gruchalski wrote: post #208

Earlier quoted context omitted.

I’ve picked up a second hand Dell R720 last year. Best purchase in years. It cost me €1500 for 48 cores and 192GB RAM.

I want one. What’s it done to your electricity bill, though?

Just checked one of ours R620, dual E5-2690 v2 @ 3.00GHz, 256Gb RAM.

20% load (9 VMs with 11GHz usage), 196W.

Historical trends for Last Week:

Avg: 203 W | 693 BTU/hr

Max: 279 W | 952 BTU/hr

Min: 150 W | 512 BTU/hr

Historical Peaks:

465 W

Cumulative Reading:

Since Mon Jul 13 11:32:33 2020 2723.893 kWh

Re: Reclaiming the lost art of Linux server administration

#310

Post by fnord77 »

senko wrote: post #160
viraptor wrote: post #139

Earlier quoted context omitted.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

> Here's what the bare metal server didn't come with: [bunch of stuff I don't need] Exactly. Imagine paying for all that when all you need is bare metal. Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do. Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy. Buy what you ne…

I'm too busy to tend to the care and feeding of a database server. I'll gladly pay amazon to do it for me.
Post reply on HN