Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

311–320 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#311

Post by MonaroVXR »

NikolaeVarius wrote: post #185
0xCMP wrote: post #94

Earlier quoted context omitted.

In my experience Pet servers are a good starting point (you really should _graduate_ from Pet servers into all the various immutable/cattle stuff), but it can quickly require discipline from the Admins. They can't be doing one-off undocumented config, package, and network/firewall changes which make it impossible to setup another server reliably. At $company I moved us to Terraform+Packer (to get them used to immutab…

I find people dont know how amazing a "immutable" server fleet is until you've experienced it. It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour. It was amazing, never had to deal with something missing on a server or a config being wrong in a special case. Dozens o…

"It was so trivial to terminate and restart dozens of servers at any given time since unless there was a mistake in the cloud-init, we could bootstrap our entire infrastructure from scratch within an hour."

Got any tips in this regard?

Ansible, Puppet, Python, Terraform, Openstack Pulumi springs to my mind. Still need to cover a PXE server. (Maybe with Ansible)

Re: Reclaiming the lost art of Linux server administration

#312

Post by MonaroVXR »

d0gsg0w00f wrote: post #300
readingnews wrote: post #75

Earlier quoted context omitted.

>>Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g…

You should learn some AWS and use it as a trojan horse to get those jobs. As a former old school sysadmin I really took to it because it's so modular and feels like Unix's "collection of simple tools piped together". Plus, any old school admin is a treasure on any cloud team because the need to drop to shell is unavoidable. People think they don't need sysadmins on cloud teams but behind every good cloud team are a f…

I want to use AWS without credit card, because I don't want to make a mistake ending up with 80K in debt. I will never forget that particular HN post.

Re: Reclaiming the lost art of Linux server administration

#313

Post by christophilus »

kirubakaran wrote: post #275
christophilus wrote: post #262

Earlier quoted context omitted.

I want one. What’s it done to your electricity bill, though?

R820s idle at ~200W and consume ~500W at 100% CPU load on all 80 virtual cores.

So, if we say it draws around 350W, that makes it around $32 /mo power (rough calculation based on a 12.5cent (US) kWh rate, right? Not bad for all that beef.

Re: Reclaiming the lost art of Linux server administration

#315

Post by geocrasher »

ryukafalz wrote: post #123
bmurphy1976 wrote: post #110

Earlier quoted context omitted.

You don't need a 42u rack. You can run a cluster of Raspberry Pi's hidden in your basement ceiling rafters like me.

Short throw from that to this classic: > hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is. http://www.bash.org/?5273

https://www.theregister.com/2001/04/12/missing_novell_server...

Re: Reclaiming the lost art of Linux server administration

#316

Post by bamboozled »

lrvick wrote: post #273
bamboozled wrote: post #187

Earlier quoted context omitted.

> I know where our data lives and no one can access it without a warrant and my explicit knowledge As far as you know? Your data is exposed to The Internet so someone could be accessing it.

You forget the wonders of TLS when you control both ends.

I'm not talking about people snooping your data, I'm talking about a configuration error or exploit which gives people access to your information.

Maybe I am naive, but I think the chances of this happening on a cloud based service like Dropbox are going to be a lot lower.

Re: Reclaiming the lost art of Linux server administration

#317

Post by willjp »

viraptor wrote: post #139
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

I'm not sure, I guess it depends what you're looking for. I'm a hobbyist, with no real professional experience in server admin, so I am probably missing some important things. But most of this can be replicated with ZFS and FreeBSD jails (or on linux, BTRFS and LXC containers).

>> 1. A solution for unlimited scheduled snapshots without affecting performance.

You can very comfortably have instant and virtually unlimited snapshots with zfs/jails (only occupying space when files change). Very easy to automate with cron and a shellscript.

>> 2. API access for managing configuration, version updates/rollbacks, and ACL.

>> 3. Close to immediate replacement of identical setup within seconds of failure.

There is a lot of choices for configuration management (saltstack, chef, ansible, ..). I run a shellscript in a cron job that takes temporary snapshots of the jail's filesystem, copies to a directory, and makes an off-site backup. A rollback is as simple as stopping the server, renaming a directory, and restarting it. It's probably more than a couple of seconds, but not by much. I think I'm uncomfortable exposing an API with root access to my systems to the internet, but I'm not sure how these systems work. I don't think it would be hard to set it up with flask if you wanted it though.

>> 4. No underlying OS management.

I don't know what this is, but I'm curious and looking it up :D.

In most of the posts I'm reading here, people have really beefy rigs. But you could do this on the cheap with a 2000s era laptop if you wanted (that was my first server).

Re: Reclaiming the lost art of Linux server administration

#318

Post by eklavya »

b112 wrote: post #293
outworlder wrote: post #265

Earlier quoted context omitted.

> Exactly. Imagine paying for all that when all you need is bare metal. Yes. The opposite is also true though: Imagine not wanting to pay for that and needing it! There's a reason why most homes connect to the power utility companies. Yes, we can run generators ourselves. Does it make sense to do that? Not usually. Same thing with this server. If it makes sense for your use-case, outstanding. In many cases, people ar…

Exactly! Who needs a kitchen in their home, it's better to just order in all the time. All that complexity of cooking daily, preparing and storing food, cleaning dishes, etc, etc. I mean, who's going to even do the cooking! See... anyone can pick , and describe how people don't do it. Of course, you cite a generator, others use solar cells, right?

Yeah imagine cutting your own hair and diagnosing your health problems. Growing your own food and making your own chips.

Looks like sensible/nonsensical based on who you are and what are your needs. Not everybody needs cloud, not everybody can self host.

Re: Reclaiming the lost art of Linux server administration

#319

Post by alexgaribay »

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I self-host my stuff as well but with a 2012 Mac Mini running Proxmox. I use it as a means to learn sysadmin things.

I get my static IP by using the smallest VPS from vultr with a wireguard tunnel forwarding http/s traffic to a docker container running nginx proxy manager.

For those wanting to learn, I highly recommend joining r/homelab and r/selfhosted. Those communities have a lot in common and you can learn a lot.

Re: Reclaiming the lost art of Linux server administration

#320

Post by willjp »

lrvick wrote: post #271
Osiris wrote: post #269

Earlier quoted context omitted.

What do you do for off-site backups?

I threw a NAS in a friends house that is configured to bootup and reverse ssh tunnel to my rack where it then accepts scheduled encrypted duplicity backups. I do not even need to trust my friend as duplicity encrypts all data against a yubikey held pgp keychain before it leaves. The backup NAS could phone home from anywhere with internet access.

This is very slick. Apparently, I need to make some techie friends. Tarsnap is pretty affordable for off-site backups though!
Post reply on HN