Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

381–390 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#381

Post by rhn_mk1 »

Nextgrid wrote: post #227
rhn_mk1 wrote: post #142

Earlier quoted context omitted.

Caution: you may end up with your packets blackholing on the way for unknown reasons after temporary loss of connectivity. I think it might have something to do with the NAT forgetting about my UDP "connection", but haven't found the culprit yet.

Could be because you’re using a terrible ISP-provided router. Switch that to bridge mode and connect the server directly (if necessary, use the server as a router with a second network card for your LAN) or use a proper, enterprise-grade router (OpenWRT would work as well as long as the underlying hardware is performant enough).

> Switch that to bridge mode

Sadly, not possible on a shared residential connection. I tried the setup where the server was also the router, and that was much less convenient in case of failures than having a dedicated router.

But yes, the abundance of routers is definitely a problem here, with an OpenWRT and a FritzBox in the mix. Perhaps I should try to get IPv6 running and forget about NAT woes.

Re: Reclaiming the lost art of Linux server administration

#382

Post by porker »

0xCMP wrote: post #94
candiddevmike wrote: post #13

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

In my experience Pet servers are a good starting point (you really should _graduate_ from Pet servers into all the various immutable/cattle stuff), but it can quickly require discipline from the Admins. They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutab…

The "missing magic" I'd like to see is a script that knows I started with Debian, installed these packages, and derives the set up script to reproduce this server including my configuration changes. That way I can make the changes in an environment that works for me and have technology do the boring part.

Re: Reclaiming the lost art of Linux server administration

#383

Post by madjam002 »

ayushnix wrote: post #324
madjam002 wrote: post #95

Honestly after discovering NixOS I have a new found joy of administering Linux servers. It's easy and painless, everything is declarative and versioned, and new machines can be set up for new projects or scaling in a matter of minutes. This "cattle not pets" mentality doesn't make sense for everything and is highly inefficient if the OS itself seamlessly supports immutable workloads and configuration.

> It's easy and painless It was probably the most difficult thing I tried to, unsuccessfully, use on my desktop. I imagine learning how to use Vim/Emacs as a complete beginner would probably be several magnitudes of order easier than learning the Nix DSL and the Nix way of doing things. And from what I've read about the experience of other people who do use NixOS and talk about both the good AND the bad, using it see…

I hear you, Nix definitely does feel slow and I must day over the few years I’ve been using it it has definitely slowed down even more.

In terms of the DSL, I’m really surprised that people find it a problem as much as they do. When I first tried Nix I moved my first nonprod server over to it that very afternoon. It helped looking at the syntax to start with as “hmm this is a bit like JSON” and to worry about things like lazy evaluation later on. I guess it helped that I was familiar with JSONNET beforehand so maybe that helped.

My first few servers I got by with just a basic understanding of the language and copying/pasting examples from the website and using https://search.nixos.org/options

Using on a desktop, now that is a whole other experience. I personally quite enjoy it for my desktop but there is definitely more of a learning experience and you might not find the benefits of Nix worth it anyway as your desktop is an always changing environment.

Re: Reclaiming the lost art of Linux server administration

#384

Post by nix23 »

akireu wrote: post #119
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

It's 2022 and we're about to rediscover something we know for 40 years already: mainframes are freaking expensive.

You misread something, it's 2022 and we discover that mainframes can be cheaper then a AWS-cloud-machine-park ;)

Re: Reclaiming the lost art of Linux server administration

#385

Post by rad_gruchalski »

axiolite wrote: post #301
christophilus wrote: post #262

Earlier quoted context omitted.

I want one. What’s it done to your electricity bill, though?

If you want to use 4Kn drives, SSDs (w/trim), or easily switch between RAID and HBA modes, I would recomend an R730 with H730 controller instead. They're available on eBay with 128GB of RAM for just over $1,000 USD. If you don't want those features, an R720 is a good choice, and around half the price. Idle power consumption of an R730 with no drives is typically about 85W, though higher-end CPUs drive that up a bit.…

Yes, R730 would have been better. I learned that post fact.

Re: Reclaiming the lost art of Linux server administration

#386

Post by _xnmw »

SkipperCat wrote: post #2

Time is money, and the more time I spend on infrastructure, the less time I spend on product. And thus is born the incredible demand of infrastructure as a service. Thankfully one person's cloud is another person's on prem infrastructure so sysadmin skills will always be in demand. From my perspective in enterprise computing, I now see people taking 2 paths. One where they become super deep sysadmins and work on infr…

This is a straw-man argument. Of course if the cloud could save me time and energy, I would use it, but it doesn't. In my experience, you spend just as much time in the long run tweaking/configuring the AWS console as you do simply running bash scripts on a baremetal server. That's why "AWS consultant" roles exist as a full-time job. The cloud does NOT save time for me, and it's far worse in many ways: opaque, more expensive, and can lull you into a false sense of security (what do you do, for example, if your cloud providers backups just fail one day?).

Re: Reclaiming the lost art of Linux server administration

#387

Post by _xnmw »

viraptor wrote: post #139
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

Using cloud-managed backups can lull you into a false sense of security. I've heard enough of horror stories of the snapshot/restore mechanism failing for some reason, especially on certain edge-cases when you use huge amounts of data. What then? All your cloud marketing magic becomes a liability.

Re: Reclaiming the lost art of Linux server administration

#388

Post by dspillett »

Aeolun wrote: post #350
dspillett wrote: post #174

Earlier quoted context omitted.

I assume/hope you have good, tested, off-site backups if the important data & config… I run my stuff from home too, though it is smaller scale than yours currently. Off-site & soft-offline backups are on encrypted volumes on servers & VMs elsewhere.

The backup server in my home is the offsite backup. The primary data lives in the datacenter.

Until recently my home server was not capable in many respects to the outside machines (picked for cheap storage, not CPU/RAM considerations, going with the "redundant array of inexpensive backup locations" method), but I do currently have a variable dedicated server that might see some live workloads moved to it.

Re: Reclaiming the lost art of Linux server administration

#389

Post by taubek »

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I'm where I'm today because of my love for Linux. In the 90s my go to distribution was Slackware. I had it spread across several dozens of floppy disks.

Knowing how to keep your server running and understanding internals is a great skill, but that doesn't mean that progress should stop.

Standalone servers are great, but this greatness comes at a price. It takes time to maintain server, it takes time to configure additional services. But at the same time they bring you joy (and frustrations) and much more knowledge and deeper understanding of what goes under the hood.

Re: Reclaiming the lost art of Linux server administration

#390

Post by _xnmw »

As a solo bootstrapped SaaS founder who literally relies on my app staying online to pay my rent and food, I chose NOT to use the cloud, even though I have 6 figures in cloud credits. I use the credit to spin up a single beefy baremetal instance and manage my own services. It's because I realized the cloud was based on a false promise which seemed possible a few years ago, but I now realized is impossible. There is no such thing as a "fully managed" deployment where you don't have to think about servers. See also: self-driving cars -- very easy to build a toy demo case, fails catastrophically in the real world. The cloud promised to save us time on managing, deploying and scaling servers; except in my experience: it doesn't. You spend just as much time, if not more time, dealing with the scaling issues of managed databases and "app engines" as you do SSH'ing into your server. Except it's worse because you lose touch with reality and have a very poor understanding of what your code is actually doing.

Last week my choice was vindicated: I ran into a critical hardware issue on my linux instance which required a complete OS reinstallation. Wiped my server clean, and was back up and running in an hour. I feel much more secure in the fact that I KNOW I can spin up a completely functional version of my app on any Linux server in the world in less than an hour, rather than relying on opaque cloud backup/load balancers/serverless configs which could fail in unexpected ways, and are usually locked in to a particular vendor. As for a few hours downtime here and there, my business is designed to handle it.

Post reply on HN