Earlier quoted context omitted.
I'm not sure, I guess it depends what you're looking for. I'm a hobbyist, with no real professional experience in server admin, so I am probably missing some important things. But most of this can be replicated with ZFS and FreeBSD jails (or on linux, BTRFS and LXC containers). >> 1. A solution for unlimited scheduled snapshots without affecting performance. You can very comfortably have instant and virtually unlimit…
> You can very comfortably have instant and virtually unlimited snapshots with zfs/jails Yes, but that both requires manual scripting it and remains local to the server. Compare to scheduled RDS backups which go to S3 with all its consistency guarantees. > There is a lot of choices for configuration management (saltstack, chef, ansible, ..) Sure, those are an improvement over doing things manually. But for the recove…
Reclaiming the lost art of Linux server administration
351–360 of 485 posts
Re: Reclaiming the lost art of Linux server administration
#352Re: Reclaiming the lost art of Linux server administration
#353Earlier quoted context omitted.
Do you have any tips in auto installing an OS on a server/desktop? I'm completely missing. I have searched arpund and I have some solutions, but back in my head, some people have something else. The part before Ansible or puppet to kick in.
NixOS.
When I wanted to set up NixOS, I wanted to use it to manage a bunch of NixOS virtual machines (so, nixops), and then run a couple of services. Here are a few points from my anecdotal experience:
1. Learn an entirely new language that has absolutely horrendous documentation. It used to be (I've been told it got better but I abandoned Nix before this) extremely difficult to debug as well, since oftentimes you would get an error in some library file when the error was obviously in your own file.
2. Try packaging your own stuff. Again, barely any documentation. One time, I wrote a simple Rails application in a day. It took me more than three days to figure out how to deploy it, and that involved just figuring it out. Rails adds a PID file to the directory with your Ruby code in it, but that directory in a Nix derivation (fancy [which makes it harder] name for something like package from my understanding) is immutable. Good luck packaging complicated projects that someone else hasn't packaged yet.
3. Nixops was so incredibly outdated that it caused my server to say it was using an "insecure" library from Python 2.7 and my auto updates started failing. On paper, nixops looked nice (manage NixOS virtual machines/VPSes with Nix). But it broke my updates. I eventually wrote my own replacement for Nixops, which worked, but I still don't trust the rest of NixOS to not break. The tool might have some merit on a different Linux distribution, though. (I'd also note that I believe that nixops fixed their insecurity bug, but there were a myriad of other issues I don't remember that I had with nixops)
4. Setting up pre-packaged services sounds easy. They were, somewhat. But permission errors were pervasive. SystemD tempfiles were finnicky at best and gave path traversal errors that were extremely hard to debug.
Eventually, I got a new server, installed Arch on it, and use docker-compose for everything. It takes me maybe 20 minutes to set up a new service on a good day, instead of four hours of Googling for some obscure error from NixOS. While not perfectly "reproducible," neither was NixOS, because I could simply not trust a virtual machine on nixops to boot successfully—there would always be more required even if it worked once.
I might be open to NixOS on a desktop, though.
Re: Reclaiming the lost art of Linux server administration
#354I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…
Re: Reclaiming the lost art of Linux server administration
#355There are plenty of books around. And there are literally thousands of people worldwide practicing this "lost" art daily.
Starting from small corp up to the major cloud providers. (Someone has to support those computers, running the "serverless" things")
My word of advice: start with the "philosophy". One program doing only one task but extremely well, "everything is a file" etc.
Understand why people are unhappy with SystemD. :-) Find out how kernel schedulers impact databases' IO. Write a boring program in C - network server which forks on accept4. Tip your toe in Perl 5 - there is lots of it in *nix and BSD. Still most stable and efficient way of writing CGI script ... Find out why Ksh is faster than Bash.
It is truly exciting world, and the best news is that it "fits" as a glove the modern world of JS and async programming etc.
I wouldn't call it "lost" - it is just dozen of levels of abstractions down, efficient, boring and complex. But powerful and unforgiving to typos :-)
I am glad someone actually is reading about all that.
Re: Reclaiming the lost art of Linux server administration
#356Earlier quoted context omitted.
> Here's what the bare metal server didn't come with: [bunch of stuff I don't need] Exactly. Imagine paying for all that when all you need is bare metal. Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do. Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy. Buy what you ne…
> Exactly. Imagine paying for all that when all you need is bare metal. Yes. The opposite is also true though: Imagine not wanting to pay for that and needing it! There's a reason why most homes connect to the power utility companies. Yes, we can run generators ourselves. Does it make sense to do that? Not usually. Same thing with this server. If it makes sense for your use-case, outstanding. In many cases, people ar…
The minute gens/solar/wind/batteries combo becomes less expensive than public utility I'll switch. For now it makes no financial sense.
With the clouds it is the other way around. My dedicated servers running my software kick the shit out of AWS performance wise for a fraction of the price. And no I do not spend my days "managing" it. I can order new dedicated server and the same shell script will reinstall all prerequisites, restore data from backup and run it in few minutes add (however long it takes to import database from backup). Where needed I also have standby up to date servers.
Other than running this script to test restoration once a month my management overhead is zero.
Re: Reclaiming the lost art of Linux server administration
#357A very weird thread that degenerated into: "PaaS vs self-hosted/self-owned hardware" . I'm pretty sure most people sysadmin'ing their Linux servers are actually doing it with rented dedicated servers. TFA btw specifically mentions: "don't manage physical hardware" . Big companies like Hetzner and OVH have hundreds of thousands of servers and they're not the only players in that space. They don't take care of "everyth…
How good is their 50 euro package? I wanna try it just for fun and dicking around.
Re: Reclaiming the lost art of Linux server administration
#358Earlier quoted context omitted.
A CDN wouldn't be enough if the traffic spike involves writes.
Oh no a website went down! Oh, wait, that's not an emergency. Where did the idea that every site and service needs five nines availability come from? A side project goes down or is read only for a few hours. Who gives a shit? It's not an insulin pump or nuclear control rods. It's ok for people to be mildly inconvenienced for a short period of time.
I guess people who use Twitter don't like seeing bad things written about them on Twitter. Though the solution is to disown Twitter and all who use it.
Re: Reclaiming the lost art of Linux server administration
#359Re: Reclaiming the lost art of Linux server administration
#360Earlier quoted context omitted.
Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…
> Here's what the bare metal server didn't come with: [bunch of stuff I don't need] Exactly. Imagine paying for all that when all you need is bare metal. Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do. Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy. Buy what you ne…
They tend to be the things that you don't need until you absolutely need them right now (or yesterday).