Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reclaiming the lost art of Linux server administration

pietrorea.com

Post reply on HN
Print view View original

71–80 of 485 posts

Re: Reclaiming the lost art of Linux server administration

#71

Post by hnthrowaway0315 »

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

This sounds very interesting. Do you have a blog describing your experience just curious? Thanks for sharing~~

Re: Reclaiming the lost art of Linux server administration

#72

Post by aledalgrande »

TacticalCoder wrote: post #62
aledalgrande wrote: post #52

Earlier quoted context omitted.

I'm just curious, because to me it seems a little bit unrealistic. How do you handle traffic spikes, especially from the networking point of view? What kind of connection do you have? How do you make your service as fast for all customers around the world (saying you have a succesful Saas). How do you prevent a local blackout from taking down your service? Where do you store your backups, in case your building gets f…

> How do you handle traffic spikes, especially from the networking point of view? I don't know about GP but managing your own server doesn't mean you cannot use a CDN with your webapp.

A CDN wouldn't be enough if the traffic spike involves writes.

Re: Reclaiming the lost art of Linux server administration

#73

Post by chasil »

ericbarrett wrote: post #65
chasil wrote: post #25

Earlier quoted context omitted.

Alias expansion within scripts is mandated by POSIX. When bash is not in POSIX mode, it violates the standard. $ ll /bin/sh lrwxrwxrwx. 1 root root 4 Nov 24 08:40 /bin/sh -> bash $ cat s1 #!/bin/sh alias p=printf p hello\\n $ cat s2 #!/bin/bash alias p=printf p world\\n $ ./s1 hello $ ./s2 ./s2: line 3: p: command not found

That's very nice, it's POSIX-compliant when invoked as #!/bin/sh and sane when invoked as #!/bin/bash — exactly what I'd want.

If you want a portable script, then you don't want that behavior.

Re: Reclaiming the lost art of Linux server administration

#74

Post by TacticalCoder »

A very weird thread that degenerated into: "PaaS vs self-hosted/self-owned hardware".

I'm pretty sure most people sysadmin'ing their Linux servers are actually doing it with rented dedicated servers. TFA btw specifically mentions: "don't manage physical hardware". Big companies like Hetzner and OVH have hundreds of thousands of servers and they're not the only players in that space.

They don't take care of "everything" but they take care of hardware failure, redundant power sources, Internet connectivity, etc.

Just to give an idea: 200 EUR / month gets you an EPYC 3rd gen (Milan) with shitloads of cores and shitloads of ECC RAM and a fat bandwith.

And even then, it's not "dedicated server vs the cloud": you can have very well have a dedicated server and slap a CDN like CloudFlare on your webapp. It's not as if CloudFlare was somehow only available to people using an "entire cloud stack" (whatever that means). It's the same for cloud storage / cloud backups etc.

I guess my point is: being a sysadmin for your own server(s) doesn't imply owning your own hardware and it doesn't imply either "using zero cloud services".

Re: Reclaiming the lost art of Linux server administration

#75

Post by readingnews »

bblb wrote: post #41
candiddevmike wrote: post #13

Blame the folks demonizing/shaming having "pet" servers and pushing immutable infrastructure. Linux server administration is quite enjoyable, and with how well apps these days can scale vertically, it really takes a special kind of workload to need (and actually saturate) fleets of servers.

I work in IT Operations of a big IT house. 100% local gov customers. We fully manage around 5000 pet servers. ~30 sysadmins and some of us do architecture designing also. There's also a separate networking team of about 10 network specialists. Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also. My hunch is that the 5000 onprem pet servers are not going away any…

>>Senior sysadmins are really hard to come by today, not to mention someone who wants to do architecture also.

I am not so sure... I am a well seasoned sysadmin, been doing server, network, architecture. I consider myself a solid linux/network expert and have managed datacenters. When I look for a new/more exciting job, or for a pay raise, all I see are "cloud, AWS, devops". I never see "old school" sysadmin jobs e.g. as you say, we have a room full of linux boxes and we manage them with ansible/scripts/etc, but we design and maintain them ourselves, come join our team".

Re: Reclaiming the lost art of Linux server administration

#76

Post by dfinninger »

rank0 wrote: post #44
warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

Do you have a static IP? I have a homelab too but getting “enterprise grade” service from comcast seems to be my biggest barrier to scaling without leaning on aws.

For a while with my home lab I cronned a python script to look up my external DNS and update my domain in Route53 (and Dyn before that). Worked out pretty well, I think it only updated once or twice in two years.

Re: Reclaiming the lost art of Linux server administration

#77

Post by bblb »

LAC-Tech wrote: post #36

Full disclaimer, I'm very much not a sysadmin or devops guy. However, every team I've been on recently has spent a lot of time struggling with gluing their AWS stuff together, diagnosing bugs etc. It didn't seem to save a heck of a lot of time at all. I couldn't figure out AWS. But I could figure out how to host sites on a linux VPS. So what's the story here - is serverless something that only makes sense at a certai…

Caddyserver is Apache/Nginx killer and we will talk about Caddy in a couple of years as if it's been always the default ("why did we kept fighting with apache/nginx all those years, silly us"). Seriously. It's just a completely different way to think about web servers and automation. I'm just amazed it took all these years to emerge.

Re: Reclaiming the lost art of Linux server administration

#78

Post by jenscow »

johnchristopher wrote: post #11
chasil wrote: post #3

-"As for scripting, commit to getting good at Bash." That advice can cause substantial headache on Ubuntu/Debian, where the Almquist shell is /bin/sh. This does not implement much of bash and will fail spectacularly on the simplest of scripts. This is also an issue on systems using Busybox. A useful approach to scripting is to grasp the POSIX shell first, then facets of bash and Korn as they are needed. -"As a practi…

I settled with using /bin/sh for portability. If there is something that can't be done with sh but can be done with bash then it means a python script is better anyway. I don't want to deal with bashism and ksh and deb/ub/rh different takes on bash. It's frustrating that most google search results and shell script search results on SO almost always mean bash and sh.

Yes, well if a script I wrote has somehow ended up on a machine without bash, then I'd be more worried about other assumptions the script makes.

Re: Reclaiming the lost art of Linux server administration

#79

Post by lamontcg »

LAC-Tech wrote: post #36

Full disclaimer, I'm very much not a sysadmin or devops guy. However, every team I've been on recently has spent a lot of time struggling with gluing their AWS stuff together, diagnosing bugs etc. It didn't seem to save a heck of a lot of time at all. I couldn't figure out AWS. But I could figure out how to host sites on a linux VPS. So what's the story here - is serverless something that only makes sense at a certai…

> is serverless something that only makes sense at a certain scale?

Other way around. With enough scale you should be able to make hosting your own datacenter work.

The problem is that the people you hire tend to go off buying too much Enterprise-class shit and Empire building and the whole thing winds up costing 10 times as much as it should because they want stuff to play with to resume stuff and to share risk with the vendor and have them to blame.

Only thing Amazon did to build out their internal IT ops exceptionally cheaply and eventually sell it as the AWS cloud service was to focus on "frugality" and fire anyone who said expensive words like "SAN". And they were ordered in no uncertain terms to get out of the way of software development and weren't allowed to block changes the way that ITIL and CRBs used to.

I didn't realize how difficult that would be to replicate anywhere else and foolishly sold all my AMZN stock options thinking that AWS would quickly get out competed by everyone being able to replicate it by just focusing on cheap horizontal scalability.

These days there is some more inherent stickiness to it all since at small scales you can be geographically replicated fairly easily (although lots of people still run in a single region / single AZ -- which indicates that a lot of businesses can tolerate outages so that level of complexity or cost isn't necessary -- but in any head-to-head comparison the "but what if we got our shit together and got geographically distributed?" objection would be raised).

Re: Reclaiming the lost art of Linux server administration

#80

Post by DizzyDoo »

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

Woah, 42u, that's quite something! How many of those slots are you using? What kind of cooling do you have to have for that closet, and how is the volume for the rest of the house?
Post reply on HN