Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS to bare metal two years later: Answering your questions about leaving AWS

oneuptime.com

Post reply on HN
Print view View original

271–280 of 513 posts

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#271

Post by tayo42 »

dangus wrote: post #108
Spooky23 wrote: post #80

Earlier quoted context omitted.

I don’t think it’s paid shilling, it’s dogma that reflects where people are working here. The individual engineers are hammers and AWS is the nail. AWS/Azure/GCP is great, but like any tool or platform you need to do some financial/process engineering to make an optimal choice. For small companies, time to market is often key, hence AWS. Once you’re a little bigger, you may develop frameworks to operate efficiently.…

I think a lot of engineers who remember the bare metal days have legitimate qualms about going back to the way that world used to work especially before containerization/Kubernetes. I imagine a lot of people who use Linux/AWS now started out with bare metal Microsoft/VMWare/Oracle type of environments where AWS services seemed like a massive breath of fresh air.

Containers with k8s and bare metal aren't mutually exclusive.

If anything it enables a hybrid environment

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#272

Post by jmaker »

fulafel wrote: post #26
bilekas wrote: post #11

I'm so surprised there is so much pushback against this.. AWS is extremely expensive. The use cases for setting up your system or service entirely in AWS are more rare than people seem to realise. Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? > We have 730+ days with 99.993% measured availability and we also escaped AWS region wide downtime…

The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things the AWS way, getting AWS certified to ensure they build your systems the AWS Well Architected Way instead of thinking themselves, and can upsell you on AWS lock-in solutions using AWS provided soundbites and sales arguments. ("Shall we make the app very resilie…

It’s a marketing trap. But also a job guarantee since everyone’s in the same trap. You got a couple cloud engineers or "DevOps" that lobby for AWS or any other hyperscaler, NaiveDate managers that write down some decision report littered with logical fallacies, and a few years in the sink cost is so high you can’t get off of it, and instead of doing productivity work you’re sitting in myriads of FinOps meetings, where even fewer understand what’s going on.

Engineering mangers are promised cost savings on the HR level. Corporate finance managers are promised OpEx for CapEx trade-off, the books look better immediately. Cloud engineers are embarking on their AWS journey of certification being promised an uptick to their salaries. It’s a win/win for everyone, in isolation, a local optimum for everyone, but the organization now has to pay way more than it—hypothetically—would have been paying for bare metal ops. And hypothetical arguments are futile.

And it lends itself well to overengineering and the microservices cargo cult. Your company ends up with a system distributed around the globe across multiple AZs per region of business operations, striving to shave off those 100ms latency off your clients’ RTT. But it’s outgrown your comprehension, and it’s slow anyway, and you can’t scale up because it’s expensive. And instead of having one problem, you now have 99 and your bill is one.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#273

Post by jmaker »

hinkley wrote: post #253
fulafel wrote: post #26

Earlier quoted context omitted.

The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things the AWS way, getting AWS certified to ensure they build your systems the AWS Well Architected Way instead of thinking themselves, and can upsell you on AWS lock-in solutions using AWS provided soundbites and sales arguments. ("Shall we make the app very resilie…

My last team decided to hand manage a Memcached cluster because it cost half as much as an unmanaged service versus AWS’s alternative. Don’t know how much we really saved versus opportunity cost on dev time though. But it’s close to negative.

One of the issues there is that pricing a managed service deprives your people or gaining extra experience. There’s a synergy over time, the more you manage yourself. But it’s totally justified to pick a managed service if it checks out for your budget. The problem I saw often emanate was bad decision making, bad opportunity cost estimation. In other words, there’s an opportunity cost to picking the managed service, too, and they offset each other more or less.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#274

Post by geoka9 »

torginus wrote: post #239
MontyCarloHall wrote: post #169

Earlier quoted context omitted.

Assuming those 20PB are hot/warm storage, S3 costs roughly $0.015/GB/month (50:50 average of S3 standard/infrequent access). That comes out to roughly $3.6M/year, before taking into account egress/retrieval costs. Does it really cost that much to maintain your own 20PB storage cluster? If those 20PB are deep archive, the S3 Glacier bill comes out to around $235k/year, which also seems ludicrous: it does not cost six…

How the heck does anyone have that much data? I once built myself a compressed plaintext library from one of those data-hoarder sources that had almost every fiction book in existence, and that was like 4TB compressed (but would've been much less if I bothered hunting for duplicates and dropped non-English). I suspect the only way you could have 20PB is if you have metrics you don't aggregate or keep ancient logs (wh…

In addition to what others have mentioned, before the "AI bubble", there was a "data science bubble" where every little signal about your users/everything had to be saved so that it could be analyzed later.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#275

Post by bryanlharris »

wredcoll wrote: post #190
whstl wrote: post #176

Earlier quoted context omitted.

I've worked on both kinds of companies in almost 25 years and I can confirm this is true. The biggest ops teams I worked alongside were always dedicated to running AWS setups. The slowest too were dedicated to AWS. Proportionally, I mean, of course. People here are comparing the worst possible of Bare Metal with "hosting my startup on AWS".

> The biggest ops teams I worked alongside were always dedicated to running AWS setups. The slowest too were dedicated to AWS. I wish I could come up with some kind of formalization of this issue. I think it has something to do with communication explosions across multiple people.

Increases in complexity exponentially increases mistakes + MS Teams meetings are just a glorified game of telephone.

Don't make perfect the enemy of the good.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#276

Post by jmaker »

torginus wrote: post #204
fulafel wrote: post #26

Earlier quoted context omitted.

The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things the AWS way, getting AWS certified to ensure they build your systems the AWS Well Architected Way instead of thinking themselves, and can upsell you on AWS lock-in solutions using AWS provided soundbites and sales arguments. ("Shall we make the app very resilie…

Unfortunately it's not, and it gets more difficult the more cloud-y your app gets. You can pay for EC2+EBS+network costs, or you can have a fancy cloud native solution where you pay for Lambda, ALBs, CloudWatch, Metrics, Secret Manager, (things you assume they would just give you, like if you eat at a restaurant, you probably won't expect to pay for the parking, toilet, or paying rent for the table and seats). So clo…

We run regular FinOps meetings within departments, so everyone’s aware. I think everyone should. But it’s a lot of overhead of course. So a dev is concerned not only with DevOps anymore but with DevSecFinOps. Not everyone can cope with so many aspects at once. There’s a lot of complexity creep in that.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#277

Post by Spooky23 »

dangus wrote: post #108
Spooky23 wrote: post #80

Earlier quoted context omitted.

I don’t think it’s paid shilling, it’s dogma that reflects where people are working here. The individual engineers are hammers and AWS is the nail. AWS/Azure/GCP is great, but like any tool or platform you need to do some financial/process engineering to make an optimal choice. For small companies, time to market is often key, hence AWS. Once you’re a little bigger, you may develop frameworks to operate efficiently.…

I think a lot of engineers who remember the bare metal days have legitimate qualms about going back to the way that world used to work especially before containerization/Kubernetes. I imagine a lot of people who use Linux/AWS now started out with bare metal Microsoft/VMWare/Oracle type of environments where AWS services seemed like a massive breath of fresh air.

No doubt -- there are plenty of downsides to running your own stuff. I'm not anti-AWS. I'm pro-efficiency, and pro making deliberate choices. If there's a choice is spend $10M extra on AWS because the engineers get a good vibe -- there should be a compelling reason why that vibe is worth $10M. (And there may well be)

Look at what Amazon/Google/Microsoft does. If you told me you advocate running your own power plants, I'd eyeroll. But... if you're as large a power consumer as a hyper-scaler, totally different story. Google and Microsoft are investing in lighting up old nuclear plants.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#278

Post by kyledrake »

The article mentions Equinix Metal but if you look it up they are shutting down the service https://docs.equinix.com/metal/hardware/standard-servers

Doesn't make me want to be a Equinix customer when they just randomly shut down critical hosting services.

I'm pretty sure that it's just the post-merger name for Packet which was an incredible provider that even had BYO IP with an anycast community. Really a shame that it went away, it was a solid alternative to both AWS and bare metal and prices were pretty good.

There's a missing middle between ultra expensive/weird cloud and cheap junk servers that I would really love to see get filled.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#279

Post by evanelias »

speleding wrote: post #267
evanelias wrote: post #208

Earlier quoted context omitted.

Why are you ignoring the huge middle ground between "HA with fully automated failover" and "no replication at all"? Basic async logical replication in MySQL/MariaDB is extremely easy to set up, literally just a few commands to type. Ditto for doing failover manually the rare times it is needed. Sure, you'll have a few minutes of downtime until a human can respond to the "db is down" alert and initiates failover, but…

I've done quite a few MySQL setups with replication. I would not call setup "extremely easy", but then, I'm not a full time DB admin. MySQL upgrades and general trouble shooting is so much more painful than AWS aurora where everything just takes a few clicks. And things like blue/green deployment, where you replicate your entire setup to try out a DB upgrade, are really hard to do onprem.

Without specifics it's hard to respond. But speaking as a software engineer who has been using MySQL for 22 years and learned administrative tasks as-needed over the years, personally I can't relate to anything you are saying here! What part of async replication setup did you find painful? How does Aurora help with troubleshooting? Why use blue/green for upgrade testing when there are much simpler and less expensive approaches using open source tools?

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#280

Post by whstl »

ghaff wrote: post #195
wredcoll wrote: post #186

Earlier quoted context omitted.

This is the root success of aws, it lets internal teams bypass sysadmin departments.

Working around official IT was certainly a significant factor early on. I'm less convinced it is nearly as big a driver (or a downside depending on your perspective) today.

Especially considering that outside of startups (where approval would be fast with or without cloud), virtual infrastructure also got its own bureaucratic process.
Post reply on HN