Live data from Hacker News

AWS to bare metal two years later: Answering your questions about leaving AWS

oneuptime.com

251–260 of 513 posts

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#251
post #72

Earlier quoted context omitted.

Unless you put someone on retainer to be responsible, which you can do cheaper than to keep your AWS setup from breaking... (I do that for people; my AWS using customers consistently end up needing more help)

The point isn't cost, it's dodging responsibility.

You can dodge responsibility equally well by outsourcing to people who'll run your bare metal setup for you. We exist from small consultancies like mine to huge multinationals.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#252
post #11

I'm so surprised there is so much pushback against this.. AWS is extremely expensive. The use cases for setting up your system or service entirely in AWS are more rare than people seem to realise. Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? > We have 730+ days with 99.993% measured availability and we also escaped AWS region wide downtime…

> I'm so surprised there is so much pushback against this.. AWS is extremely expensive. I see more comments in favor than pushing back. The problem I have with these stories is the confirmation bias that comes with them. Going self-hosted or on-premises does make sense in some carefully selected use cases, but I have dozens of stories of startup teams spinning their wheels with self-hosting strategies that turn into…

> I have dozens of stories of startup teams spinning their wheels with self-hosting strategies that turn into a big waste of time and headcount that they should have been using to grow their businesses instead.

Funnily enough, the article even affirms this, though most people seemed to have skimmed over it (or not read it at all).

> Cloud-first was the right call for our first five years. Bare metal became the right call once our compute footprint, data gravity, and independence requirements stabilised.

Unless you've got uncommon data egress requirements, if you're worried about optimizing cloud spend instead of growing your business in the first 5 years you're almost certainly focusing on the wrong problem.

> You really need to weigh the tradeoffs, but many people are not equipped to do that. They just think their chosen solution will be perfect and the other side will be the bad one.

This too. Most of the massive AWS savings articles in the past few days have been from companies that do a massive amount of data egress i.e. video transfer, or in this case log data. If your product is sending out multiple terabytes of data monthly, hosting everything on AWS is certainly not the right choice. If your product is a typical n-tier webapp with database, web servers, load balancer, and some static assets, you're going to be wasting tons of time reinventing the wheel when you can spin up everything with redundancy & backups on AWS (or GCP, or Azure) in 30 minutes.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#253
post #26
post #11

I'm so surprised there is so much pushback against this.. AWS is extremely expensive. The use cases for setting up your system or service entirely in AWS are more rare than people seem to realise. Maybe I'm just the old man screaming at cloud (no pun intended) but when did people forget how to run a baremetal server ? > We have 730+ days with 99.993% measured availability and we also escaped AWS region wide downtime…

The direct cost is the easy part. The more insidious part is that you're now cultivating a growing staff of technologists whose careers depend on doing things the AWS way, getting AWS certified to ensure they build your systems the AWS Well Architected Way instead of thinking themselves, and can upsell you on AWS lock-in solutions using AWS provided soundbites and sales arguments. ("Shall we make the app very resilie…

My last team decided to hand manage a Memcached cluster because it cost half as much as an unmanaged service versus AWS’s alternative. Don’t know how much we really saved versus opportunity cost on dev time though. But it’s close to negative.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#254
post #9

It's an interesting article, thanks for that. What people forget about the OVH or Hetzner comparison is that for those entry servers they are known for, think the Advance line with OVH or AX with Hetzner. Those boxes come with some drawbacks. The OVH Advance line for example comes without ECC memory, in a server, that might host databases. It's a disaster waiting to happen. There is no option to add ECC memory with t…

Their current advance offerings use AMD EPYC 4004 with on-die ECC. I can’t figure out if it’s “real” single correction double detection, or if the data lines between the processor and dimms are protected or not though.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#255
Recently i learned that orgs these days want to show software and infrastructure spend as capex as they can shown it as depreciating asset for tax purposes.

I understand that with AWS you cannot do that as it is often seem as opex.

I guess thats a good enough motivation to move out of AWS at scale.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#256

Earlier quoted context omitted.

I'd say the core of their success is running everything in a single rack in a single datacenter at first (for months? a year?) and getting lucky. Life is simple when you don't need the costs and effort of reliability upfront.

They mention having a second half-rack in a different DC. In any case, not everyone need five nines, and usually it's just much easier to bring down a platform due to some bug in your own software rather that the core infrastructure going down at a rack level.

The point is valid, they mention adding that, so at one point they didn't have that. They're also only storing monitoring & observability data, that's never going to be mission critical for their customers.

It's probably the main reason why they were able to get away with this and why their application does not need scalability. I see they themselves are only offering two 9s of uptime.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#257

Earlier quoted context omitted.

> I'm so surprised there is so much pushback against this.. AWS is extremely expensive. I see more comments in favor than pushing back. The problem I have with these stories is the confirmation bias that comes with them. Going self-hosted or on-premises does make sense in some carefully selected use cases, but I have dozens of stories of startup teams spinning their wheels with self-hosting strategies that turn into…

> The shared theme of all of the failure stories is missing the true cost of self-hosting: The hours spent getting the servers just right, managing the hosting, debating the best way to run things, and dealing with little issues add up but are easily lost in the noise if you’re not looking closely. What the modern software business seems to have lost is the understanding that ops and dev are two different universes.…

> DevOps was a reaction to the fact that even outsourcing ops to AWS doesn’t entirely solve all of your ops problems and the role is absolutely no substitute for a systems administrator.

This is revisionist history. DevOps was a reaction to the fact that many/most software development organizations had a clear separation between "developers" and "sysadmins". Developers' responsibility ended when they compiled an EXE/JAR file/whatever, then they tossed it over the fence to the sysadmins who were responsible for running it. DevOps was the realization that, huh, software works between when the people responsible for building the software ("Dev") are also the same people responsible for keeping it running ("Ops").

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#258

This is a completely meaningless article if they don't provide information about their technical stack, which AWS services they used to use, what TPS they are hitting, what storage size they're using, etc. The story will be different for every business because every business has different needs. Given the answer to "How much did migration and ongoing ops really cost?" it seems like they had an incredibly simple infra…

TFA begins with a link to the original article with those details.

If you called "We used EKS" details, then yeah they provide those details.

Assuming this is indeed all they used, this was admittedly nonsense, they were essentially using cloud-based bare-metal.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#259
post #9

It's an interesting article, thanks for that. What people forget about the OVH or Hetzner comparison is that for those entry servers they are known for, think the Advance line with OVH or AX with Hetzner. Those boxes come with some drawbacks. The OVH Advance line for example comes without ECC memory, in a server, that might host databases. It's a disaster waiting to happen. There is no option to add ECC memory with t…

These concerns are exaggerated. I've been running on Hetzner, OVH and friends for 20 years. During that time I've had only two issues, one about 15 years ago when a PSU failed on one of the servers, and another a few years ago when an OVH data center caught fire and one of the servers went down. There have been no other hardware issues. YMMV.

They matter at scale, where 1% issues end up happening on a daily or weekly basis.

For a startup with one rack in each of two data centers, it’s probably fine. You’ll end up testing failover a bit more, but you’ll need that if you scale anyway.

If it’s for some back office thing that will never have any load, and must not permanently fail (eg payroll), maybe just slap it on an EC2 VM and enable off-site backup / ransomware protection.

Re: AWS to bare metal two years later: Answering your questions about leaving AWS

#260

> AWS is extremely expensive. I really like how people throw around these baseless accusations. S3 is one of the cheapest storage solutions ever created. The last 10 years I have migrated roughly 10-20PB worth of data to AWS S3 and it resulted in significant cost saving every single time. If you do not know how to use cloud computing than yes, AWS can be really expensive.

Assuming those 20PB are hot/warm storage, S3 costs roughly $0.015/GB/month (50:50 average of S3 standard/infrequent access). That comes out to roughly $3.6M/year, before taking into account egress/retrieval costs. Does it really cost that much to maintain your own 20PB storage cluster? If those 20PB are deep archive, the S3 Glacier bill comes out to around $235k/year, which also seems ludicrous: it does not cost six…

Like most of AWS, it depends if you need what it provides. A 20PB tape system will have an initial cost in the low to mid 6 figures for the hardware and initial set of tapes. Do the copies need to be replicated geographically? What about completely offline copies? Reminds me of conversations with archivists where there's preservation and then there's real preservation.
Post reply on HN