Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS to bare metal two years later: Answering your questions about leaving AWS

oneuptime.com

Post reply on HN
Print view View original

441–450 of 513 posts

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#441

Post by sgarland »

parliament32 wrote: post #343
esskay wrote: post #54

Earlier quoted context omitted.

> I'm so surprised there is so much pushback against this I'm not. It seems to be happening a lot. Any time a topic about not using AWS comes up here, or on Reddit there a sudden surge of people appearing out of nowhere shouting down anyone who suggests other options. It's honestly starting to feel like paid shilling.

I don't think it's paid shilling, I think it's people who got bamboozled into learning cloud-provider-clickops over actual systems work and feel threatened when you suggest hyperscalers aren't the future.

Nailed it. I’ve found that people get super defensive when you inadvertently reveal that they don’t know the fundamentals of their job.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#442

Post by sgarland »

chickensong wrote: post #372
Aurornis wrote: post #153

Earlier quoted context omitted.

> I'm so surprised there is so much pushback against this.. AWS is extremely expensive. I see more comments in favor than pushing back. The problem I have with these stories is the confirmation bias that comes with them. Going self-hosted or on-premises does make sense in some carefully selected use cases, but I have dozens of stories of startup teams spinning their wheels with self-hosting strategies that turn into…

All valid and important points, but missing a painful one, also rarely represented in threads like this: flaky hardware. Almost every bare metal success story paints a rosy picture of perfect hardware (which thankfully is often the case), or basic hard failures which are easily dealt with. Disk replacement or swapping 1u compute nodes is expected and you probably have spares on hand. But it's a special feeling to deb…

Not saying those things don’t happen, but having worked with on-prem for 2 years, and having ran ancient (13 years old currently) servers in my homelab for 5 years, I’ve never seen them. Bad CPU, bad RAM, yes - and modern servers are extremely good at detecting these and alerting you.

In my homelab, in 5 years of running the aforementioned servers (3x Dell R620, and some various Supermicros) 24/7/365, the only thing I had fail was a power supply. Turns out they’re redundant, so I ordered another one, and the spare kept the server up in the meantime. If I was running these for a business, I’d keep hot spares around.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#443

Post by sgarland »

wredcoll wrote: post #180
DrewADesign wrote: post #172

Earlier quoted context omitted.

> The shared theme of all of the failure stories is missing the true cost of self-hosting: The hours spent getting the servers just right, managing the hosting, debating the best way to run things, and dealing with little issues add up but are easily lost in the noise if you’re not looking closely. What the modern software business seems to have lost is the understanding that ops and dev are two different universes.…

> What the modern software business seems to have lost is the understanding that ops and dev are two different universes. This is a fascinating take, if you ask me, treating them as separate is the whole problem! The point of being an engineer is to solve real world problems, not to live inside your own little specialist world. Obviously there's a lot to be said for being really good at a specialized set of skills, b…

The issue is that precious few devs are actually good at ops. There are a ton of abstractions that have sprung up that attempt to paper over this, and they all suck for various reasons.

If you think you need to actually know your programming language of choice to be a good dev, I have news for you about actually knowing how computers work to be good at ops.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#444

Post by chasemp »

Edman274 wrote: post #191
hibikir wrote: post #115

Earlier quoted context omitted.

Well, they aren't wrong about the bare metal either: Every organization ends up tied to their staff, and said staff was hired to work on the stack you are using. People end up in quite the fights because their supposed experts are more fond of uniformity and learning nothing new. Many a company was stuck with a datacenter unit that was unresponsive to the company's needs, and people migrated to AWS to avoid dealing w…

The entire value proposition of AWS vs running one's own server is basically this: is it easier to ask for permission, or forgiveness? You're asking for permission to get a million dollars worth of servers / hardware / power upgrades now, or you're asking for forgiveness for spending five million dollars in AWS after 10 months. Which will be easy: permission or forgiveness?

I had not thought of it this way, but interesting point. I have seen this as well.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#445

Post by sgarland »

nostrebored wrote: post #425
whstl wrote: post #176

Earlier quoted context omitted.

I've worked on both kinds of companies in almost 25 years and I can confirm this is true. The biggest ops teams I worked alongside were always dedicated to running AWS setups. The slowest too were dedicated to AWS. Proportionally, I mean, of course. People here are comparing the worst possible of Bare Metal with "hosting my startup on AWS".

This is a toupee situation. Every effective company I've worked at has a slim platform team that might make some nice company specific templates for how to deploy, but individual teams were responsible for creating and owning their infra. The idea of having an AWS ops team is absurd if you're not at a truly massive company (XX,000+)

I have never, ever seen dev-created infra that was well done, much less with repeatable IaC. It’s always résumé-driven nonsense based on whatever someone read on blogs, and they have no clue how any of it works, only that the output what they expect.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#446

Post by sgarland »

torginus wrote: post #214
thelastgallon wrote: post #175

These are the features that AWS provides (1) Massive expansion of budget (100 - 1000x) to support empire building. Instead of one minimum-wage sysadmin with 2 high-availability, maxed-out servers for 20K - 40K (and 4-hour response time from Dell/HPE), you can have 100M multi-cloud Kubernetes + Lambda + a mix-and-match of various locked-in cloud services (DB, etc.). And you can have a large army of SRE/DevOps. You get…

The problem with those 5 people, is you can't hire a 6th - your stack is custom and probably even if you find the guy, he'll need months of ramp-up. In contrast, you could throw a stone into a bush and hit an AWS guy.

Bare-metal management isn’t custom, it’s just not cloud. A TFTP server, your choice of confirmation management, your choice of image builder, probably - but not necessarily - a hypervisor, and that’s pretty much it.

This stuff isn’t hard, and it was solved well over a decade ago. Probably two at this point. It’s just been shoved down by the hyperscalers (who themselves are busily hiring for those skills).

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#447

Post by tayo42 »

canucktrash669 wrote: post #416
tayo42 wrote: post #269

Earlier quoted context omitted.

What size companies are we talking about

I've seen this in startups with 7 figure ARR (where annual cloud costs were also 7 figures). Also seen that in F500 where a single architect caused a 5-figure mistake which remove cloud privileges from the entire architecture team. Can't make it up.

I mean 5 figures is nothing to most companies when your spending 10s of millions on aws.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#448

Post by geodel »

jmaker wrote: post #383
geodel wrote: post #375

Earlier quoted context omitted.

All great points. I have seen in company of smart people CIO/CTO would freely up admit "Look we know cloud may not be cheap or easier to manage but this is the direction we have taken since we are getting out of owning or managing hardware/datacenter" So it is not like one can dazzle decision makers with any logic or hard data. They are just announcing the decision while calling it a robust discussion over pros and c…

Yep. I’ve also seen managerial people worship AWS sales reps as oracles, misconstruing ordinary sales meetings with them as something divine, in which they would disclose a lot of company’s IP in awe for them, just to listen to some blabbing superficial truisms. I mean, ChatGPT could tell you more. To add insult to that, the managerial people wouldn’t listen to their own senior, staff, principal engineers, and prefer…

> For my peace of mind, I chose a sane path - if the company as an entity decides to do AWS, I will do my best to meet its goals...

Amen to that.

Any kind of performance improvement, monitoring work I did for my applications has met with indifference or derision from managers. Because only if I had put efforts in cloud migration we could be "Horizontal Pod Scaling" for performance and fully managed Datadog console for monitoring the services.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#449

Post by [deleted] »

Aeolun wrote: post #414
alemanek wrote: post #330

Earlier quoted context omitted.

In orgs I have seen this it is usually a symptom of the data center unit being starved of resources. It’s like they have only been given the choice of on prem but ridiculous paperwork and long lead times or pay 20x for cloud. Like can’t we just give the data center org more money and they can over provision hardware. Or can we not have them use that extra money to rent servers from OVH/Hetzner during the discovery ph…

I feel like companies are unreasonably afraid of cost up front, never mind that they’re going to pay more for cloud over the next 6 months, spending 6x monthly cloud cost on a single server makes them hesitate. It’s how they always refuse to spend half my monthly salary on the computer I work on, and instead insist I use an underpowered windows machine.

[deleted]

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#450

Post by winrid »

hinkley wrote: post #253
fulafel wrote: post #26

Earlier quoted context omitted.

The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things the AWS way, getting AWS certified to ensure they build your systems the AWS Well Architected Way instead of thinking themselves, and can upsell you on AWS lock-in solutions using AWS provided soundbites and sales arguments. ("Shall we make the app very resilie…

My last team decided to hand manage a Memcached cluster because it cost half as much as an unmanaged service versus AWS’s alternative. Don’t know how much we really saved versus opportunity cost on dev time though. But it’s close to negative.

That should be extremely low maintenance...
Post reply on HN