Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

61–70 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#61

Post by lrvick »

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem.

When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber.

I know where our data lives and no one can access it without a warrant and my explicit knowledge. I also save myself several hundred a month in third party cloud provider fees to host the same services and can reboot upgrade or repair anything whenever I want, but in general no more maintenance than cloud servers . I also never end up with exciting bills when experiments are forgotten about.

You pretty much get all the pros and cons of home ownership. For me it is mostly pros. Also keeps me dogfooding all the same practices I recommend to my clients.

Re: Reclaiming the lost art of Linux server administration

#62

Post by TacticalCoder »

aledalgrande wrote: post #52
warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

I'm just curious, because to me it seems a little bit unrealistic. How do you handle traffic spikes, especially from the networking point of view? What kind of connection do you have? How do you make your service as fast for all customers around the world (saying you have a succesful Saas). How do you prevent a local blackout from taking down your service? Where do you store your backups, in case your building gets f…

> How do you handle traffic spikes, especially from the networking point of view?

I don't know about GP but managing your own server doesn't mean you cannot use a CDN with your webapp.

Re: Reclaiming the lost art of Linux server administration

#63

Post by dijit »

deepspace wrote: post #56
warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has…

You need to replicate in Cloud too, most people tend not to because they think the cloud is magic, but it's computers and computers can fail- even if they're someone else's.

Also "if some component fails or the app is critical" has a lot of nuance, I agree with your sentiment but you should know:

1) Component failures in hardware are much rarer than you think

2) Component failures in hardware can be mitigated (dead ram, dead PSU, dead hard disk, even dead CPUs in some cases: all mitigated) The only true failure of a machine is an unmitigated failure due to not configuring memory mirroring or something' or a motherboard failure (which is extremely uncommon)

3) The next step after "single server" isn't "build a datacenter", it's buying a couple more servers and renting half a rack from your local datacenter, they'll have redundant power, redundant cooling and redundant networking. They'll even help you get set up if it's 2-3 machines with their own hardware techs.

I do this last one at a larger scale in Bahnhof.

also, $3000 will get you about 3-5 years out of hardware, at which point, yeah, you should think about upgrading, if for no other reason than it's going to be slower.

Re: Reclaiming the lost art of Linux server administration

#64

Post by marcosdumay »

jhickok wrote: post #40
marcosdumay wrote: post #32

Earlier quoted context omitted.

> Entirely eliminating "system administration" from my duties... ... and adding "cloud administration". What is it with people doing completely one-sided analysis even when they experiment the thing by themselves? Is cloud administration less time consuming than system administration? That's not my experience, so I'm quite interested on how it got so.

Can be. Paying for SaaS offerings like the gsuite or o365 is a great deal for say, 100 seats, instead of paying someone to administer on prem email. "Cloud administration" can be more work, less work, or about the same work as classic systems administration. That's why carefully running the numbers first should be necessary.

Oh, sure. Offshoring email is much easier than running it yourself.

The same isn't true for less standard kinds of service. The more standardized something is the easiest it is to decide what to hire, troubleshoot, and learn to configure your options. The less standardized it is, the harder all of those things become. VMs are very standard, email servers are less so, but not by a huge margin. Web accessible disk space and on-demand interpreters are completely non-standardized and a hell to do anything with.

Also, some services do need more upkeep than others. Email is one extreme that requires constant care, file storage and web servers demand much less attention.

Re: Reclaiming the lost art of Linux server administration

#65

Post by ericbarrett »

chasil wrote: post #25
ericbarrett wrote: post #15

Earlier quoted context omitted.

Not sure what you are saying, bash behaves as bash when invoked as /bin/bash, and Bourne-shell-ish when invoked as /bin/sh. Lots more detail in the man page. I've never seen use of aliases in a bash script...? They are generally for CLI convenience.

Alias expansion within scripts is mandated by POSIX. When bash is not in POSIX mode, it violates the standard. $ ll /bin/sh lrwxrwxrwx. 1 root root 4 Nov 24 08:40 /bin/sh -> bash $ cat s1 #!/bin/sh alias p=printf p hello\\n $ cat s2 #!/bin/bash alias p=printf p world\\n $ ./s1 hello $ ./s2 ./s2: line 3: p: command not found

That's very nice, it's POSIX-compliant when invoked as #!/bin/sh and sane when invoked as #!/bin/bash — exactly what I'd want.

Re: Reclaiming the lost art of Linux server administration

#66

Post by peterbabic »

MattGaiser wrote: post #48
LAC-Tech wrote: post #36

Full disclaimer, I'm very much not a sysadmin or devops guy. However, every team I've been on recently has spent a lot of time struggling with gluing their AWS stuff together, diagnosing bugs etc. It didn't seem to save a heck of a lot of time at all. I couldn't figure out AWS. But I could figure out how to host sites on a linux VPS. So what's the story here - is serverless something that only makes sense at a certai…

A lot of it is lack of awareness of things like Caddy or any other tools that simplify the process. I did not know about it until I googled it right now. I have spent days/even two weeks figuring out how to set up Nginx and for all I know I did it terribly wrong. I paired it with other tools that I do not even remember. But I would be starting from scratch again if I needed to set another one up. So a lot might come…

Even with Caddy, there are so many rabbit holes ro get down into. My current one is rootless. I feel like in completely different world compared to rootfull. Learned a ton though

Re: Reclaiming the lost art of Linux server administration

#67

Post by jefurii »

culopatin wrote: post #58

Since the post pretty much says “go out and do it!” Does anyone have a good source of learning that is comprehensive and practical? I’m talking about a good guided book/tutorial on how to administer a server properly and what things one should know how to fix, not just how to set up Wordpress.

When I learned this stuff I started with The Debian Administrator's Handbook (https://debian-handbook.info) and an O'Reilly book called Unix Power Tools. Since then I've read handbooks for whatever servers/frameworks/apps I needed to use. There was never one single source. I've also spent lots of time googling error messages and reading Stack Overflow/Server Fault and other sites.

Re: Reclaiming the lost art of Linux server administration

#69

Post by chasil »

gh02t wrote: post #57
chasil wrote: post #18

Earlier quoted context omitted.

Bash has a POSIX mode. Knowing when to switch into and out of this mode, and what impact it has, is a more advanced subject that should not burden those learning the Borne family. It is better to start with Almquist, or another pure POSIX implementation, with documentation specific to standard adherence. More advanced shell features should wait.

I have mixed feelings. On paper I agree with you, people should start with POSIX shell. In practice I'm not sure how relevant that is anymore. I'm not really convinced bash should be the default thing people learn, but I think there's a decent argument that people should just start off with the shell they're going to actually use. You should , however, be aware that there is a distinction, and if you're learning/writ…

For me, I just want a shell that works.

Without a nuanced understanding of standards, extensions, and platform availability, new bash users will get large amounts of shell usage that doesn't work.

To avoid that frustration, learn POSIX. That works everywhere that matters.

Re: Reclaiming the lost art of Linux server administration

#70

Post by cpach »

culopatin wrote: post #58

Since the post pretty much says “go out and do it!” Does anyone have a good source of learning that is comprehensive and practical? I’m talking about a good guided book/tutorial on how to administer a server properly and what things one should know how to fix, not just how to set up Wordpress.

This is a good start: https://www.ansiblefordevops.com/
Post reply on HN