Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

151–160 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#151

Post by chasil »

notyourday wrote: post #117
chasil wrote: post #3

-"As for scripting, commit to getting good at Bash." That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. This does not implement much of bash and will fail spectacularly on the simplest of scripts. This is also an issue on systems using Busybox. A useful approach to scripting is to grasp the POSIX shell first, then facets of bash and Korn as they are needed. -"As a practi…

> That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. #!/bin/bash There, I fixed your "what shell is /bin/sh" problem.

Unless you actually look at real Linux deployments, which are:

  #!/bin/mksh
Android doesn't allow GPL code in userland, and the installed base is massive.

Re: Reclaiming the lost art of Linux server administration

#152

Post by c2h5oh »

lnxg33k1 wrote: post #144
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Just out of curiousity but what did you get out of it? I mean if they get to pick it, get the responsibility of AWS on you, but do you also get to get paid those money after saving them money? I mean the way I see it, is that 16k/month don't only pay for the hardware, but also to keep headaches away and to have someone to blame

IIRC I got to move some heavy analytics queries, which could not be ran on a read only replica (rollups) and without hearing the word budget once. Main db remained on RDS.

Re: Reclaiming the lost art of Linux server administration

#153

Post by LoveGracePeace »

TacticalCoder wrote: post #74

A very weird thread that degenerated into: "PaaS vs self-hosted/self-owned hardware" . I'm pretty sure most people sysadmin'ing their Linux servers are actually doing it with rented dedicated servers. TFA btw specifically mentions: "don't manage physical hardware" . Big companies like Hetzner and OVH have hundreds of thousands of servers and they're not the only players in that space. They don't take care of "everyth…

I generally agree, I have a cheap AWS Lightsail VPS (mainly for email hosting since my ISP blocks port 25 because I'm a "consumer" and they want to protect the world for consumer spammers) but also for flexibility. I like that the Internet is not at my doorstep (no open ports at home). So, cheap VPS, Wireguard and my home machines to serve whatever I want. I don't pay extra if I use a ton of CPU or disk IO, for example.

Here is my Wireguard server (cheap VPS) and client (my home servers) config:

# # Client (the actual self-host local server) #

[Interface] ## This Desktop/client's private key ## PrivateKey = redacted

## Client ip address ## Address = 10.10.123.2/24

[Peer] ## Ubuntu 20.04 server public key ## PublicKey = redacted

## set ACL ## AllowedIPs = 0.0.0.0/0

## Your Ubuntu 20.04 LTS server's public IPv4/IPv6 address and port ## Endpoint = redacted:12345

## Key connection alive ## PersistentKeepalive = 15

# # Server (in the Wireguard context, exposed to the Internet) #

[Interface] ## My VPN server private IP address ## Address = 10.10.123.1/24

## My VPN server port ## ListenPort = 12345

## VPN server's private key i.e. /etc/wireguard/privatekey ## PrivateKey = redacted

PostUp = iptables -i eth0 -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.10.123.2 # Add lines for more ports if desired

PostDown = iptables -i eth0 -t nat -D PREROUTING -p tcp --dport 80 -j DNAT --to-destination 10.10.123.2 # Add lines for more ports if desired

[Peer] ## Desktop/client VPN public key ## PublicKey = redacted

## client VPN IP address (note the /32 subnet) ## AllowedIPs = 10.10.123.2/32

Re: Reclaiming the lost art of Linux server administration

#154

Post by [deleted] »

candiddevmike wrote: post #118
notyourday wrote: post #116

Earlier quoted context omitted.

> Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers. You don't need pet servers. Puppet or Ansible make your baremetal cattle.

I think most folks would argue "cattle" means using imaging to manage/replace your fleet. Using something like puppet or ansible against a fresh install implies a level of individualism towards each system as they "may" have minute details based on when puppet/ansible ran, even if they're part of a dynamic inventory of some sort.

[deleted]

Re: Reclaiming the lost art of Linux server administration

#155

Post by theodric »

deepspace wrote: post #56
warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has…

There's a big difference in service criticality between your business website and your NAS full of pirated tentacle hentai. Cases like the latter can accept extended outages, and are very cost-effectively served by home-based infra.

Re: Reclaiming the lost art of Linux server administration

#156

Post by wwweston »

StillBored wrote: post #85
deepspace wrote: post #56

Earlier quoted context omitted.

> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has…

So, just buy another and leave it as a hot (or cold) standby in a different data-center. Or use AWS as the DR site an spin it up only if the local HW fails. This sounds expensive if your talking one server and vs a year of AWS charges, but is a tiny bump if it turns out you need to buy a dozen servers to replace a large AWS bill. Plus, I think most people underestimate how reliable server grade hardware is. Most of i…

> Or use AWS as the DR site an spin it up only if the local HW fails.

Yep. This seems like the obvious setup to me:

1) make the usual case as economical as possible (and ownership and the associated control will probably help here, unless you have to lease the expertise too)

2) outsource the exceptional case (ownership is less likely to matter here, and will matter for less time even if it does)

Re: Reclaiming the lost art of Linux server administration

#157

Post by belter »

c2h5oh wrote: post #87
lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Not a fair comparison and you know it. Now add to the $200 month bare metal server, the yearly salary of the 3 admins you need to manage it. One as backup, one for day time, one for the night plus a weekend rotation. Add to the admin salaries social security, insurance and a margin of safety if one is unavailable due to sickness.

Re: Reclaiming the lost art of Linux server administration

#158

Post by Eduard »

wrt https://gist.github.com/pietrorea/9081e2810c20337c6ea85350a3... :

Don't use "here documents" or "here strings" for passwords. Even in bash versions as recent as 2020, they create a temporary file within `/tmp/` with the secret inside. If the timing is unlucky, it will get written to disk and therefore leave permanent traces even after reboot. Only shredding will securely delete the data.

Re: Reclaiming the lost art of Linux server administration

#159

Post by variant »

Not quite the angle the author was getting at, but have noticed at $dayjob that staff who are able to do some incredibly complex automation against Linux-based stacks, containers, etc. - get quite lost when something low level isn't working right. Gaps in understanding of OS level troubleshooting and concepts gets them stuck.

You're wise to keep staff around who understand the low level stuff, in addition to the shiny new abstraction based tools.

Re: Reclaiming the lost art of Linux server administration

#160

Post by senko »

viraptor wrote: post #139
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

> Here's what the bare metal server didn't come with:

[bunch of stuff I don't need]

Exactly. Imagine paying for all that when all you need is bare metal.

Now imagine paying for all that just because you've read on the Internet that it's best practice and that's what the big guys do.

Way back the best practice was what Microsoft, Oracle or Cisco wanted you to buy. Now it's what Amazon wants you to buy.

Buy what you need.

Post reply on HN