Why I recommended ECS instead of Kubernetes to my latest customer • Hacker News Board

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#91

Post by alien_ » Thu, Jun 08, 2023, 6:22 PM UTC

itsmemattchung wrote: post #2

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

The OP here, thanks for your comment.

To be honest I wasn't hired to challenge their entire setup, only to make it more cost effective.

So I chose the most straightforward way I could think of that would allow us to come up with a cost effective setup that will be scalable, fault tolerant and simple to maintain later on.

It all probably started with such a single instance running Docker compose, but then over time it evolved into this setup.

The ideal setup I mentioned would have been also cost effective, scalable and resilient.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#92

Post by CharlesW » Thu, Jun 08, 2023, 6:22 PM UTC

mvdtnz wrote: post #85

CharlesW wrote: post #27

Earlier quoted context omitted.

> …the cloud is total overkill for small startups and hobby projects. It absolutely can be, sure. But solutions like Vercel, Cloudflare Workers, Supabase, etc. can be excellent and inexpensive for those use cases.

Vercel is a lot of things but I don't think I've ever seen it referred to as "inexpensive".

So far, for my hobby projects it's basically free.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#93

Post by fnordpiglet » Thu, Jun 08, 2023, 6:32 PM UTC

nova22033 wrote: post #90

fnordpiglet wrote: post #51

Earlier quoted context omitted.

As someone who sat in on the product development discussions at aws about EKS the internal view was K8S was: * a lock in strategy by Google to substitute for the fact they don’t yet have systemic abstractions at a provider level. By “owning” the design and engineering around k8s through capture they can ensure the backing services they build in gcp naturally support k8s users as they develop their roadmap * the provi…

Eventually EKS was built to satisfy customers that insisted these issues were just FUD from aws to lock customers into the aws infrastructure. I mean..the customers are not wrong.

At the most charitable minimum that wasn’t the spirit of the convos internally though. These were the points of why aws didn’t think k8s was a great idea, even for customers, if they’re customers of aws rather than gcp. Aws makes money if you use them using k8s or ECS, and once you use any stateful service or spend the time to specify the eks infrastructure, you’ve got a switching cost no matter what.

My thought in this space is go with whatever is the least effort. There is no meaningful portability between cloud providers using anything right now. But if you don’t make your stuff baroque it’s also not hard to port between one provider and another from an infrastructure specification point of view. I think the “lock in” at the specification of infrastructure is a canard. Lock in happens to a much deeper level at the integrations between dependencies inside the customers own infrastructure and the stored state. Having 1000 services across an enterprise integrated inside (aws|gcp|azure|oracle|on prem) makes it hard to switch anywhere else from a basic connectivity, rights, identity, etc level - so hard that it degenerates into why “hybrid” cloud infrastructures basically fail. But that means switching is either all or nothing, which is impractical, or you bite off this integration problem, which is apparently impossible or at least absurdly hard. Then you’re also left with stored state, which is heavy and difficult to move, let alone expensive, but also the challenge of moving the state over with the state managing services without downtime or loss of data is also pretty hard. Hard enough that you can’t expect every team owning the 1000 services can do it.

So, you can pick k8s and run an abstraction on an abstraction, or not, but when it comes time to break your lockin, k8s won’t buy you anything.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#94

Post by dilyevsky » Thu, Jun 08, 2023, 6:37 PM UTC

topspin wrote: post #59

I'm tracking cloud-hypervisor and kata containers closely. I'm convinced there is a unicorn opportunity here for the SME/private-cloud world. An easily managed cluster of lightweight, live-migratable, hardware isolated VMs running containers (as opposed to just herding containers) solves problems people actually have, as opposed to the problems k8s solves. k8s is fine for the scale of enterprise for which it is actua…

Kata container is just OCI-compatible kvm so what business need does it solve for your general acme corp that a standard docker/containerd/crio container doesn’t?

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#95

Post by HatchedLake721 » Thu, Jun 08, 2023, 6:47 PM UTC

djbusby wrote: post #80

HatchedLake721 wrote: post #76

Earlier quoted context omitted.

What SQS has to do with EC2? One is a queuing service, the other one is a VM. So instead of using SQS that has $0 cost when there are no customers, you suggest I install, configure and run RabbitMQ on an EC2, to save $0 when there are no customers? Or save $1 when I have 100 customers? SQS is dirt cheap. The point of SQS or any other usage-based AWS _developer_ service compared to DIY is that you can be up and runnin…

The question is: what are queueing for zero customers?

If I'm building a marketing automation app that allows customers to do a newsletter blast, I'll put those 1000 email recipients into a queue and run through it at a required pace with a retry interval if anything fails.

What do you suggest I do before I get my first customer?

- Blast 1000 emails in one go and pray upstream accepts it?

- Push these to a database and keep checking it with a CRON?

- Run RabbitMQ on an EC2 and push 1000 messages there?

- Implement SQS in "15 minutes" at $0 cost?

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#96

Post by mikeravkine » Thu, Jun 08, 2023, 6:52 PM UTC

CharlesW wrote: post #92

mvdtnz wrote: post #85

Earlier quoted context omitted.

Vercel is a lot of things but I don't think I've ever seen it referred to as "inexpensive".

So far, for my hobby projects it's basically free.

This is exactly how the serverless guys "get you": Low traffic is nearly free but you pay for it on the slope of the scale cost ramp.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#97

Post by 0xEFF » Thu, Jun 08, 2023, 6:53 PM UTC

danenania wrote: post #50

intelthrow6 wrote: post #21

Earlier quoted context omitted.

I don’t see the reasoning. A startup that outgrows an EC2 server will be making enough money to hire more people to scale the system properly than what was initially designed: trading away everything for development velocity. Kubernetes is not the right tool for this startup. Kubernetes is what large, old-school non-tech companies use to orchestrate resources, because it’s easier to find someone that “knows k8s” (no…

Most startups are at least going to want to be able to deploy, scale up or down, and restart an app without downtime. I wouldn't say that's overkill. While it's not impossible to do with a single instance, you can spend a lot of time shaving that yak. It's reasonable to pay a bit more to have that stuff handled for you in a robust way.

These reasons related to deployment, but there's also lots of value in the security aspects of the control plane.

  * automatic service account for each workload
  * automatic service to service auth to 3rd party services
  * the audit log
  * role based access control
  * well defined api
  * the explain subcommand
  * liveness and readiness probes
  * custom resources

The list goes on, but the big ones for a small team just getting started are workload identity and security.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#98

Post by alien_ » Thu, Jun 08, 2023, 7:32 PM UTC

lapser wrote: post #33

I find it really wild that anyone would ever recommend ECS. A developer deploying a service involves: - Setting up certs (managed as TF) - Setting up ALBs (managed as TF) - Setting up the actual service definition (often done as a JSON, that is passed into TF) Possibly other things I'm forgetting. Some other things. It requires a *developer* to know about certs and ALBs and whatever else. With EKS, this can all be au…

The OP here, thanks for the comment!

Why does the developer need to care about the certs and ALBs? The devops engineer you need to set up all those controllers could as well deploy those resources from Terraform.

As I showed in the diagrams from the article this application has a single ALB and a single cert per environment and the internal services only talk to each other through the rabbit MQ queue.

DNS, ALB and TLS certs could be easily handled from just a few lines of Terraform, and nobody needs to touch it ever again.

With EKS you would need multiple controllers and multiple annotations controlling them, and then each controller will end up setting up a single resource per environment.

The controllers make sense if you have a ton of distinct applications sharing the same clusters, but this is not the case here, and would be overkill.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#99

Post by Nextgrid » Thu, Jun 08, 2023, 7:33 PM UTC

HatchedLake721 wrote: post #76

itsmemattchung wrote: post #2

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

What SQS has to do with EC2? One is a queuing service, the other one is a VM. So instead of using SQS that has $0 cost when there are no customers, you suggest I install, configure and run RabbitMQ on an EC2, to save $0 when there are no customers? Or save $1 when I have 100 customers? SQS is dirt cheap. The point of SQS or any other usage-based AWS _developer_ service compared to DIY is that you can be up and runnin…

The AMQP protocol alone and its various, good client libraries (compared to terrible AWS SDK which is a very thin abstraction over just sending/parsing raw JSON off the wire) is by itself enough to justify RabbitMQ.

> when I need queueing functionality to increase my developer velocity so I can focus on building value rather than wasting my life installing, configuring and running anything on EC2.

SQS still requires configuration, which means you either need to use the (terrible) AWS console UI or spin up a whole Terraform/CloudFormation/CDK/etc stack, not to mention that merely connecting to it requires correctly setting up AWS IAM (so you don't use a key that gives access to your entire AWS account). Vim'ing the RabbitMQ config file in contrast doesn't seem so bad, and even just using a static hardcoded password means the worst an attacker can do is take down your queue instead of taking over your entire cloud infra.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#100

Post by danpalmer » Thu, Jun 08, 2023, 7:37 PM UTC

jorams wrote: post #29

danpalmer wrote: post #7

More accurately: "Given using AWS as a requirement, I recommended ECS instead of K8s". It's not really surprising that AWS's K8S setup isn't great, and their own implementation ties in more closely with other services they offer. It's lock-in. AWS provides just enough K8S to tick the box on a spec sheet, but have little incentive to go beyond that.

Since the post only mentions Kubernetes once , I don't really understand why it's in the title at all. > The team didn't have much DevOps expertise in-house, so a Kubernetes setup, even using a managed service like EKS, would have been way too complex for them at this stage, not to mention the additional costs of running the control plane which they wanted to avoid. The control plane cost makes sense, but I can't ima…

Also the control plane cost is basically irrelevant at any real scale, I think it's pretty much there to discourage hobby projects from taking up a free VM.