Live data from Hacker News

AWS to bare metal two years later: Answering your questions about leaving AWS

oneuptime.com

291–300 of 513 posts

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#291
post #280
post #195

Earlier quoted context omitted.

Working around official IT was certainly a significant factor early on. I'm less convinced it is nearly as big a driver (or a downside depending on your perspective) today.

Especially considering that outside of startups (where approval would be fast with or without cloud), virtual infrastructure also got its own bureaucratic process.

A lot of people forget that, when server virtualization was still gaining momentum in a lot of circles, it wasn't uncommon at less technically savvy customers--say a regional bank at the time--to be told that it might take 2 months to provision a new server.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#292
post #200

Earlier quoted context omitted.

I wonder how vibe coding will impact this. You can easily get your service up by asking claude code or whatever to just do it It produces aws yaml that’s better than many devops people I’ve worked with. In other words, it absolutely should not be trusted with trivial tasks, but you could easily blow $100K’s per year for worse.

I've been contemplating this a lot lately, as I just did code review on a system that was moving all the AWS infrastructure into CDK, and it was very clear the person doing it was using an LLM which created a really complicated, over engineered solution to everything. I basically rewrote the entire thing (still pairing with Claude), and it's now much simpler and easier to follow. So I think for developers that have d…

To me it's clear that most Ops engineers are vibe coding their scripts/yamls today.

The time difference between having a script ready has decreased dramatically in the last 3 years. The amount of problems when deploying the first time has also increased in the same period.

The difference between the ones who actually know what they're doing and the ones who don't is whether they will refactor and test.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#293
post #291
post #280

Earlier quoted context omitted.

Especially considering that outside of startups (where approval would be fast with or without cloud), virtual infrastructure also got its own bureaucratic process.

A lot of people forget that, when server virtualization was still gaining momentum in a lot of circles, it wasn't uncommon at less technically savvy customers--say a regional bank at the time--to be told that it might take 2 months to provision a new server.

I don't think anyone is forgetting that in this thread, as there's dozens of answers mentioning this.

But as an example: It took about 3 months to provision an AWS server in a recent company I consulted for due to their own bureaucracy and ineptitude of the Ops team.

On the other hand, when I needed a few CI servers for a startup I worked at, I just collected them from AppleStore during lunch hour.

Now this above is what people are "forgetting" and don't want to listen to.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#294
Ok so this may be a dumb question...but now do you handle ISP outages due to storms and stuff with on prem solutions? I'd imagine large datacenters have much more sophisticated and reliable internet connections than say an Xfinity business customer, but maybe that's wrong.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#295

I had a problem figuring out why the place I was working wanted to move from in-house to AWS; their workload was easily handled by a few servers, they had no big bursts of traffic, and they didn't need any of the specialized features of AWS. Eventually, I realized that it was because the devs wanted to put "AWS" on their resumes. I wondered how long it would take management to catch on that they were being used as a…

with "dev wanting X" nothing happens. "leadership deciding X" then it needs to get done.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#296

They were running for a long time (months? over a year?) on a single rack in a single datacenter. Eventually they scaled out but the word is eventually. I think that summarizes both sides of this debate in a nutshell. You can move off of AWS but unless you invest a lot you will take on increased risk. Maybe you'll get lucky and your one rack won't burn down. Maybe you won't. They did get lucky.

> Maybe you'll get lucky and your one rack won't burn down

Given the rates of fires in DCs, you'd rather need to be quite unlucky for it to happen to you.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#297

Earlier quoted context omitted.

My manager wants me to make this silly AWS certification. Let me go on a tangent about trains. In Spain before you board a high-speed train you need to go though full security check, like on an airport. In all other EU countries you just show up and board, but in Spain there's the security check. The problem is that even though the security check is an expensive, inefficient theatre, just in case something does blow…

> In all other EU countries you just show up and board, but in Spain there's the security check Just for curiosity's sake, did any other EU countries have any recent terrorist attacks involving bombs on trains in the capital, or is Spain so far alone with this experience?

Checkout Madrid 2004 terror attacks... So deadly that Spain left Afghanistan and Iraq afik.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#299
Running EKS on AWS was their problem. If they didn't run EKS on AWS, they would've had a considerably simpler setup running Amazon Linux, not having to upgrade Kubernetes every 3 quarters, managing network security using security groups instead of having open internal networking, and running in a single AZ would've eliminated intra-AZ costs. In large data centers like us-east-1, an individual AZ is actually internally striped for extra redundancy, and you are much more likely to experience regional downtime than single AZ downtime, especially if you have a stable workload and do not rely on tech beyond rock-solid basics (EC2, VPC, ELB, S3, EBS). If you're willing to operate a single bare metal rack in a DC, you should be willing to run in a single AWS AZ.

I don't know how much time they spend configuring/dealing with Kubernetes, but I bet it's a large chunk of the 24 hour engineer-hours per quarter. But this is not a required expense: "EKS had an extra $1,260/month control-plane fee". Running EKS adds a massive IAM policy maintenance overhead, whereas a non-EKS (EC2 w/ golden AMIs) setup results in drastically simpler IAM policies.

NAT gateways are ~$50 a month, plus data transfer. Setting up a gateway VPC endpoint to S3 will avoid having to pay transfer charges to S3.

They were at 90% reservation capacity, so they should be using reservations for greater savings and in fact, running stable workloads with reservations is something that AWS excels at. Reservation means that you will be able to terminate and re-launch instances even when there's a spike in demand from other users--your instance capacity is guaranteed.

Running the basics on VMs also effectively avoids vendor lock-in. Every cloud provider supports VMs with a RedHat clone, VPCs, load balancing, networked storage, access controls, object storage and a fixed size fleet with auto-relaunch on instance failure.

With a consistent workload, they would have very likely escaped the downtime from AWS a week ago as well, because, as per AWS, "existing EC2 instances that had been launched prior to the start of the event remained healthy and did not experience any impact for the duration of the event".

With Terraform and automation for building launchable images, you can stand up a cluster quickly in any region with secure networking, including in a separate AWS account, in the same region, for the sake of testing.

With AWS, you can set up automatic EBS backups of all your data to snapshots trivially, and even send them to a 3rd locked-down account, so they can't be accidentally wiped.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#300
post #263

I have seen multiple startups paying thousands of dollars a month in AWS bills to run a tiny service which could trivially run on an $800 desktop on a residential internet connection. It's absolutely tragic.

That’s like $24K a year. Assuming they have working failover and business continuity plans, it’s actually a really good deal (vs having a 10-20% time employee deal with it).

[deleted]
Post reply on HN