Reclaiming the lost art of Linux server administration • Hacker News Board

Re: Reclaiming the lost art of Linux server administration

#131

Post by warent » Fri, Jan 28, 2022, 9:17 PM UTC

kxrm wrote: post #101

deepspace wrote: post #56

Earlier quoted context omitted.

> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has…

I am not the guy you replied to, but I also self host my web apps. I think every project is different and not all projects demand near 100% uptime. I certainly strive for HA for my projects but at the appropriate budget and my users understand. If you are trying to go commercial you might have a different attitude but for those of us who do this mostly for fun and for some donations on the side, over complicating our…

This is an important point. My customers don't love outages (who does?) but I've had them and it doesn't really hurt that badly. My products aren't that critical. They're understanding as long as you communicate.

Re: Reclaiming the lost art of Linux server administration

#132

Post by mark242 » Fri, Jan 28, 2022, 9:18 PM UTC

demosito666 wrote: post #122

mark242 wrote: post #100

Earlier quoted context omitted.

> Is cloud administration less time consuming than system administration? Infinitely, and if you look at it from a startup lens it only makes sense. One needs to point only at the recent log4j incident. This is obviously a gigantic black swan event, but even just ongoing security patching at the OS level can be a full-time gig. There is absolutely no substitution for being able to ship code to a platform that just ru…

> in the future, 100% of the code that you write will be business logic The present reality of Lambda is quite different though. Even though the code of the function itself is more or less "business logic" (although this is a really meaningless term when we're talking about known programming languages and computers), the scaffolding around it with Terraform/CloudFormation/Serverless/etc. is substantial, riddled with…

Of course -- you're still writing code to delete a key in Elasticache within your lambda. You're writing yaml code for deployments. Hence the "in the future" portion of this slide.

The scale-to-X and scale-to-zero features of Lambda, along with the guaranteed interface to your lambda with predictable input and output requirements, is incredibly empowering for an engineering team. I can absolutely guarantee that we have spent far, far, far less time maintaining our infrastructure than what we would need to be doing if we had a big-buncha-EC2 setup.

Imagine that the environment issues get taken care of, because Amazon has teams and teams and teams of engineers who are working on just that. Cloudflare has the zero-cold-start isolates. All these platforms are heavily invested in making your development and deployment experience as easy as it can be. Concentrate on writing your code, and you'll reap the benefits.

Re: Reclaiming the lost art of Linux server administration

#133

Post by bcrosby95 » Fri, Jan 28, 2022, 9:18 PM UTC

StillBored wrote: post #85

deepspace wrote: post #56

Earlier quoted context omitted.

> I can invest in a $3000 PowerEdge server with much better hardware And when some component of the server fails, your app is unavailable until you can repair it. So you need another server for redundancy. And a load balancer. And a UPS. And a second internet connection. If your app is at all critical, you need to replicate all of this at a disaster recovery site. And buy/run/administer DR software. And hardware has…

So, just buy another and leave it as a hot (or cold) standby in a different data-center. Or use AWS as the DR site an spin it up only if the local HW fails. This sounds expensive if your talking one server and vs a year of AWS charges, but is a tiny bump if it turns out you need to buy a dozen servers to replace a large AWS bill. Plus, I think most people underestimate how reliable server grade hardware is. Most of i…

Yeah. We run servers into the ground where I work. We have around 20 of them. Average age is around 11 years old. Oldest is around 18.

Re: Reclaiming the lost art of Linux server administration

#134

Post by nickdothutton » Fri, Jan 28, 2022, 9:19 PM UTC

It is partly for this reason that I’m funding an invite-only community shell/pubnix system.

Re: Reclaiming the lost art of Linux server administration

#135

Post by warent » Fri, Jan 28, 2022, 9:20 PM UTC

rank0 wrote: post #44

warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

Do you have a static IP? I have a homelab too but getting “enterprise grade” service from comcast seems to be my biggest barrier to scaling without leaning on aws.

I have a static IP address / gigabit thru centurylink

Re: Reclaiming the lost art of Linux server administration

#136

Post by pier25 » Fri, Jan 28, 2022, 9:22 PM UTC

I know how to setup a basic VPS with firewalls, nginx, etc, but I'm scared to death to do that for a production environment available online.

Re: Reclaiming the lost art of Linux server administration

#137

Post by c2h5oh » Fri, Jan 28, 2022, 9:22 PM UTC

papito wrote: post #111

c2h5oh wrote: post #87

Earlier quoted context omitted.

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

LOL. Priceless. Having these skills is very valuable. Us old farts used to do a lot with what today would be called "bootstrapped". Scarcity is no longer a "problem", except that it is. Scarcity keeps you sharp, efficient, on the edge - where you need to be. It's also - cheaper.

Who would have guessed having local, low latency, high iops drives would be better than VM using iSCSI-attached drive for storage, right? ;-)

Re: Reclaiming the lost art of Linux server administration

#138

Post by jiggawatts » Fri, Jan 28, 2022, 9:23 PM UTC

baybal2 wrote: post #39

warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

> it pays for itself in less than a year. https://news.ycombinator.com/item?id=13198157 On one meeting we had a typical discussion with ops guys: - "why wouldn't we optimise our hardware utilisation by doing things a, b, and c." - "hardware is crap cheap these days. If you need more capacity, just throw more servers at that" - "is $24k a month in new servers crap cheap by your measure?" - "comparatively to the amount…

I’ve found this to be an unsuccessful approach in practice.

Performance is a complex, many-faceted thing. It has hidden costs that are hard to quantify.

Customers leave in disgust because the site is slow.

No amount of “throwing more cores at it” will help if there’s a single threaded bottleneck somewhere.

Superlinear algorithms will get progressively worse, easily outpacing processor speed improvements. Notably this is a recent thing — single threaded throughout was improving exponentially for decades so many admins internalised the concept that simply moving an app with a “merely quadratic” scaling problem to new hardware will always fix the problem. Now… this does nothing.

I’ve turned up at many sites as a consultant at eyewatering daily rates to fix slow apps. Invariably they were missing trivial things like database indexes or caching. Not Redis or anything fancy like that! Just cache control headers on static content.

Invariably, doing the right thing from the beginning would have been cheaper.

Listen to Casey explain it: https://youtu.be/pgoetgxecw8

You need to have efficiency in your heart and soul or you can’t honestly call yourself an engineer.

Learn your craft properly so you can do more with less — including less developer time!

Re: Reclaiming the lost art of Linux server administration

#139

Post by viraptor » Fri, Jan 28, 2022, 9:23 PM UTC

c2h5oh wrote: post #87

lrvick wrote: post #61

I have over 20 years of Linux/FreeBSD sysadmin experience ranging from universities to major silicon valley companies in both cloud and on-prem. When it comes to companies I mostly support cloud these days but when it comes to me and my family I accept every downside and host as almost all of our digital lives in a 42u rack in a gutted closet in our house with static IPs and business fiber. I know where our data live…

I remember how surprised people were when I demoed a $200/month bare metal server outperforming by a huge margin RDS MySQL instance that they were paying something upwards of 16k/month. IIRC we ended up using it as a disposable replica for some non-real time but heavy operations.

Here's what the bare metal server didn't come with:

API access for managing configuration, version updates/rollbacks, and ACL.

A solution for unlimited scheduled snapshots without affecting performance.

Close to immediate replacement of identical setup within seconds of failure.

API-managed VPC/VPN built in.

No underlying OS management.

(Probably forgot a few...) I get that going bare metal is a good solution for some, but comparing costs this way without a lot of caveats is meaningless.

Re: Reclaiming the lost art of Linux server administration

#140

Post by warent » Fri, Jan 28, 2022, 9:24 PM UTC

aledalgrande wrote: post #52

warent wrote: post #34

When my SaaS app started scaling, I saw how badly cloud can be priced if you have even slightly unusual use-cases. It occurred to me that instead of spending ~$600/mo on GCP, I can invest in a $3000 PowerEdge server with much better hardware, run it out of my home office, and it pays for itself in less than a year. Running your own server is an investment that doesn't make sense for everyone. If you can get it, it is…

I'm just curious, because to me it seems a little bit unrealistic. How do you handle traffic spikes, especially from the networking point of view? What kind of connection do you have? How do you make your service as fast for all customers around the world (saying you have a succesful Saas). How do you prevent a local blackout from taking down your service? Where do you store your backups, in case your building gets f…

CenturyLink provides me gigabit internet on a business connection. I get traffic spikes of ~100/rps and it's no problem. Could probably easily handle another order of magnitude or two. Local blackouts are mitigated with a UPS https://en.wikipedia.org/wiki/Uninterruptible_power_supply

To be fair, I'm not 100% off the cloud. Backups are on an hourly snapshot thru Restic https://restic.net/ and stored in Google Cloud Storage off-prem in case of catastrophes. Also, my Postgres database is hosted in Cloud SQL because frankly I'm not feeling experienced enough to try hosting a database myself right now.

It's really not as unrealistic as most people seem to think. People have been building online businesses for years without the cloud. Believing it's suddenly not possible is just their marketing going to work for them making them new customers imo.