Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

171–180 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#171

Post by sandworm101 »

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I'm in a similar boat. I had a server in a closet but switched to a NAS a few years ago and haven't looked back. I do run into people who thinks NAS is old hat, that everything should be backed to a cloud somewhere. These are the people who think they are watching 4k because they select that option on youtube. I want to watch 4k without buffering, without artifacts. NAS lets me do that. And when I go away for a few days or weeks, everything is turned off. No data leaks possible when the "spinning rust" isn't spinning.

Re: Reclaiming the lost art of Linux server administration

#172

Post by Nextgrid »

viraptor wrote: post #139
c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

A lot of these aren’t as important for something that’s fairly static and can’t even be autoscaled or resized live (excluding Aurora - I’m talking about standard RDS).

No access to the underlying OS can actually be a problem. I had a situation where following a DB running out of disk space it ended up stuck in “modifying” for 12 hours, presumably until an AWS operator manually fixed it. Being able to SSH in to fix it ourselves would’ve been much quicker.

Re: Reclaiming the lost art of Linux server administration

#173

Post by oceanplexian »

pmarreck wrote: post #46
warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

Does it have a backup schedule (and did you prove your restore process works)? Is it replicated to another physically-offsite location? Do you have to manage your own security keys? Load balancing? Multi region availability? How do you admin it remotely? Firewalled? Notifications of emergency situations like low disk space, downages, over-utilization of bandwidth, memory leakage, SMART warnings, etc.? What's your ver…

> Does it have a backup schedule (and did you prove your restore process works)? Is it replicated to another physically-offsite location? Do you have to manage your own security keys? Load balancing? Multi region availability? How do you admin it remotely? Firewalled? Notifications of emergency situations like low disk space, downages, over-utilization of bandwidth, memory leakage, SMART warnings, etc.? What's your version upgrade strategy? What's your OS upgrade strategy? Failover? IPv6? VPN access? DMZ?

So yes, for those of us who have done Systems Administration as a lifestyle/career, yeah you do all of those things and it's part of the fun. I started doing OS upgrades, monitoring, firewalls, and home backups of my own Linux Servers some time in High School. Over-utilization of bandwidth isn't really a "problem" unless you're doing something weird like streaming video, a 1Gbps circuit can support thousands upon thousands of requests per second.

Re: Reclaiming the lost art of Linux server administration

#174

Post by dspillett »

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I assume/hope you have good, tested, off-site backups if the important data & config…

I run my stuff from home too, though it is smaller scale than yours currently. Off-site & soft-offline backups are on encrypted volumes on servers & VMs elsewhere.

Re: Reclaiming the lost art of Linux server administration

#175

Post by exdsq »

warent wrote: post #131
kxrm wrote: post #101

Earlier quoted context omitted.

I am not the guy you replied to, but I also self host my web apps. I think every project is different and not all projects demand near 100% uptime. I certainly strive for HA for my projects but at the appropriate budget and my users understand. If you are trying to go commercial you might have a different attitude but for those of us who do this mostly for fun and for some donations on the side, over complicating our…

This is an important point. My customers don't love outages (who does?) but I've had them and it doesn't really hurt that badly. My products aren't that critical. They're understanding as long as you communicate.

Plus they still happen on AWS (or other critical bits like GitHub) so you’re not immune anyway

Re: Reclaiming the lost art of Linux server administration

#176

Post by milesvp »

viraptor wrote: post #165
c2h5oh wrote: post #147

Earlier quoted context omitted.

Of course there are a lot of benefits of using hosted databases. I like hosted databases and use them for both work and personal projects. What I have a problem with is: - the premium over bare metal is just silly - maximum vertical scaling being a rather small fraction of what you could get with bare metal - when you pay for a hot standby you can't use it as a read only replica (true for AWS and GCP, idk about Azure…

> when you pay for a hot standby you can't use it as a read only replica (true for AWS I'm not sure what you mean here. At least for MySQL you can have an instance configured as replica + read-only and used for reads. Aurora makes that automatic / transparent too with a separate read endpoint.

A hot standby is not a read replica. It's a set of servers running in a different Availability zone mirroring current prod, that is configured to automatically fail over to if the primary is offline. It's been a few years since I personally set this up in AWS, but at the time, those servers were completely unavailable to me, and basically doubled the cost of my production servers.

The fact that a hot standby is usually in some sort of read-replica state prior to failing over is a technical detail that AWS sort of tries to abstract away I think.

Re: Reclaiming the lost art of Linux server administration

#177

Post by oceanplexian »

28304283409234 wrote: post #33
akira2501 wrote: post #7

Earlier quoted context omitted.

That's when it clicked for me.. comparing my hourly salary rate vs. the cost of running these services "in the cloud." Entirely eliminating "system administration" from my duties was absolutely a net win for me and our team.

Counterintuitively, engineers that run their own servers and infra tend to gain a deeper understanding of what it takes to provide an actual running service to end users. And therefor they write software or at least there is better teamwork with "devops" infra folks. This is off course the highly subjective meaning of a greybeard unixadmin.

It's also a competitive advantage. Look at Backblaze, their business model simply wouldn't be possible on a cloud provider.

Re: Reclaiming the lost art of Linux server administration

#178

Post by notyourday »

chasil wrote: post #151
notyourday wrote: post #117

Earlier quoted context omitted.

> That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. #!/bin/bash There, I fixed your "what shell is /bin/sh" problem.

Unless you actually look at real Linux deployments, which are: #!/bin/mksh Android doesn't allow GPL code in userland, and the installed base is massive.

> Android doesn't allow GPL code in userland, and the installed base is massive.

You aren't administering Android devices.

Stop obsessing about writing portable scripts. Write scripts for the targets that you are going to run them on.

Re: Reclaiming the lost art of Linux server administration

#179

Post by [deleted] »

Nextgrid wrote: post #172
viraptor wrote: post #139

Earlier quoted context omitted.

Here's what the bare metal server didn't come with: API access for managing configuration, version updates/rollbacks, and ACL. A solution for unlimited scheduled snapshots without affecting performance. Close to immediate replacement of identical setup within seconds of failure. API-managed VPC/VPN built in. No underlying OS management. (Probably forgot a few...) I get that going bare metal is a good solution for som…

A lot of these aren’t as important for something that’s fairly static and can’t even be autoscaled or resized live (excluding Aurora - I’m talking about standard RDS). No access to the underlying OS can actually be a problem. I had a situation where following a DB running out of disk space it ended up stuck in “modifying” for 12 hours, presumably until an AWS operator manually fixed it. Being able to SSH in to fix it…

[deleted]

Re: Reclaiming the lost art of Linux server administration

#180

Post by exdsq »

aledalgrande wrote: post #52
warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

I'm just curious, because to me it seems a little bit unrealistic. How do you handle traffic spikes, especially from the networking point of view? What kind of connection do you have? How do you make your service as fast for all customers around the world (saying you have a succesful Saas). How do you prevent a local blackout from taking down your service? Where do you store your backups, in case your building gets f…

In my first job as a system tech for an IT company in 2014 we had a backup process run at 17:30 and whichever admin left last would take the backup HDD with them home lol. It worked! There was also onsite redundancy with replicated windows servers in an office across the street, which was enough. Simpler times even just 8 years ago!
Post reply on HN