Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AWS to bare metal two years later: Answering your questions about leaving AWS

oneuptime.com

Post reply on HN
Print view View original

461–470 of 513 posts

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#461

Post by darkwater »

dumbledoren wrote: post #407
darkwater wrote: post #48

The core of this success is this, IMO: > Our workload is 24/7 steady. We were already at >90% reservation coverage; there was no idle burst capacity to “right size” away. If we had the kind of bursty compute profile many commenters referenced, the choice would be different. Which TBH applies to many, many places, even if they are not aware of it.

Nah. They could have just overprovisioned to hell for much cheaper. Boxes at Hetzner cost up to 10 times less than equal level of AWS compute. Just overprovision for cheaper. You have to overprovision on the cloud anyway - you cant risk your users waiting 1-2 minutes until your new nodes/pods come up. So 'cloud is good for spiky load' argument is just a lie we tell ourselves.

Well in cloud you do over provision a bit by setting autoscaling rules in a way that you still have spare capacity while the new resources are bootstrapping.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#462

Post by sarchertech »

chasd00 wrote: post #131
calgoo wrote: post #61

Earlier quoted context omitted.

Also, when something breaks, you are responsible. If you put it in AWS like everyone else and it breaks, then its their problem not yours. We will still implement workarounds and fixes when it happens, but we are not responsible. Basic enterprise rules these days is to always pay someone else to be responsible.

> then its their problem not yours this is the main advantage of cloud, no one cares if the site/service/app is down as long as it's someone else's fault and responsibility.

I don’t know. Hours into the recent AWS outrage (when it was already being covered by mainstream news) everyone at work was running around with their hair on fire trying to figure out how to get our services back up. I even saw a 700 person Teams meeting.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#463

Post by vidarh »

nostrebored wrote: post #425
whstl wrote: post #176

Earlier quoted context omitted.

I've worked on both kinds of companies in almost 25 years and I can confirm this is true. The biggest ops teams I worked alongside were always dedicated to running AWS setups. The slowest too were dedicated to AWS. Proportionally, I mean, of course. People here are comparing the worst possible of Bare Metal with "hosting my startup on AWS".

This is a toupee situation. Every effective company I've worked at has a slim platform team that might make some nice company specific templates for how to deploy, but individual teams were responsible for creating and owning their infra. The idea of having an AWS ops team is absurd if you're not at a truly massive company (XX,000+)

The "AWS ops team" is often 1-2 people and often part of the dev team formally, often augmented by external consultancies like mine. You start seeing that kind of structure in teams with 10+ people when the dev created infra starts collapsing under its own weight and they realise they need people with actual ops skillset.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#464

Post by anal_reactor »

kleiba wrote: post #206
hedora wrote: post #189

Earlier quoted context omitted.

The security check has nothing to do with protecting trains or passengers, so your question is irrelevant.

Thanks for letting me know that my question is irrelevant. Sorry for taking up your time.

You're welcome

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#465

Post by Hilift »

infecto wrote: post #379
Hilift wrote: post #374

Earlier quoted context omitted.

Multiple Active Result Sets (MARS). During large query responses or bulk loads, "full" packets cause an additional packet to be sent over the wire with about five bytes to hold the MARS "wrapper". The net result is one full packet, and one empty packet on the wire, alternating. The performance impact in LAN latency is negligible. However on higher latency between AWS and your premises it has a terrible performance im…

Is that not a client connection flag? MARS does not require a setting change on the server?

It is a setting in connection string in the client app.config, "MultipleActiveResultSets=true;".

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#466

Post by suralind »

Love this write up. But also love the fact that the company pretty much uses the best solutions that they can (that definitely helps with hosting, regardless of cloud/self-host, but is impressive anyway). I’ve been a big Talos fan for couple years, but never worked with a client that would use something different than managed offering from their cloud provider.

Similarly, the fact that they are regularly updating Kubernetes, Talos and presumably other things means that these things are a normal flow and thus can optimize and automate. If you’re doing that because AWS is forcing you to upgrade… then it’s stressful.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#467

Post by Ragnarork »

aeve890 wrote: post #90

>We're now moving to Talos. We PXE boot with Tinkerbell, image with Talos, manage configs through Flux and Terraform, and run conformance suites before each Kubernetes upgrade. Gee, how hard is to find SE experts in that particular combination of available ops tools? While in AWS every AWS certified engineer would speak the same language, the DIY approach surely suffers from the lack of "one way" to do things. Change…

> Gee, how hard is to find SE experts in that particular combination of available ops tools?

You find expert in Ops, not in tools. People that know the fundamentals, not just the buttons to push in "certain situations" without knowing what's really going on under the hood.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#468

Post by chickensong »

sgarland wrote: post #442
chickensong wrote: post #372

Earlier quoted context omitted.

All valid and important points, but missing a painful one, also rarely represented in threads like this: flaky hardware. Almost every bare metal success story paints a rosy picture of perfect hardware (which thankfully is often the case), or basic hard failures which are easily dealt with. Disk replacement or swapping 1u compute nodes is expected and you probably have spares on hand. But it's a special feeling to deb…

Not saying those things don’t happen, but having worked with on-prem for 2 years, and having ran ancient (13 years old currently) servers in my homelab for 5 years, I’ve never seen them. Bad CPU, bad RAM, yes - and modern servers are extremely good at detecting these and alerting you. In my homelab, in 5 years of running the aforementioned servers (3x Dell R620, and some various Supermicros) 24/7/365, the only thing…

I'm glad it's working for you! It's worked for me in the past as well, but I've also felt the pain. As I mentioned before, it's often the case that things will work, but in some ways, you need to have an increased appetite for risk.

I suppose it depends on scale and requirements. A homelab isn't very relevant IMHO, because the sample size is small and the load is negligible. Push the hardware 24/7 and the cracks are more likely to appear.

A nice-to-have service can suffer some downtime, but if you're running a non-trivial/sizable business or have regulation requirements, downtime can be rough. Keeping spare compute servers is normal, but you'll be hard pressed to convince finance to spend big money on core services (db, storage, networking) that are sitting idle as backups.

Say you convinced finance to spend

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#469

Post by wombarly »

titanomachy wrote: post #333
montecarl wrote: post #301

Earlier quoted context omitted.

I can't believe how affordable Hetzner is. I just rented a bare metal 48 core AMD EPYC 9454P with 256 GB of ram and two 2 TB NVME ssds for $200/month (or $0.37 per hour). Its hard to directly compare with AWS, but I think its about 10x cheaper.

Wow. Probably performs better too, with a recent CPU and non-"elastic" disk. What about ingress/egress?

Hetzner dedicated servers by default have 1 GBit and free egress. You can opt for 10 GBit, which has 20TB and then €1 / TB overage.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#470

Post by stack_framer »

My question is: How did you get your entire team on board with this decision? My team disagrees on even the most trivial technicalities, so I can't imagine doing something on the scale you're doing (even though I wish we could leave AWS for our own hardware).
Post reply on HN