Why I recommended ECS instead of Kubernetes to my latest customer • Hacker News Board

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#41

Post by ecshafer » Thu, Jun 08, 2023, 4:50 PM UTC

itsmemattchung wrote: post #2

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

With so many developers I feel that there is a complete lack of familiarity with what it takes to just run a website. So many came up in the land of cloud and k8s and etc. There are use cases for these more advanced production environments. But if more developers just learned how to make a website on linux, with a db, a webserver, and an application. They would know that a lot of more complex things just aren't needed... especially when you don't even have customers.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#42

Post by tedivm » Thu, Jun 08, 2023, 4:50 PM UTC

lapser wrote: post #33

I find it really wild that anyone would ever recommend ECS. A developer deploying a service involves: - Setting up certs (managed as TF) - Setting up ALBs (managed as TF) - Setting up the actual service definition (often done as a JSON, that is passed into TF) Possibly other things I'm forgetting. Some other things. It requires a *developer* to know about certs and ALBs and whatever else. With EKS, this can all be au…

I honestly don't understand where you're coming from. If a devops engineer can set things up on eks for people to launch without thinking of those things, what's stopping that same engineer from doing similar for ecs?

When I was at Rad AI we went with ECS. I made a terraform module that handled literally everything you're talking about, and developers were able to use that to launch to ECS without even having to think about it. Developers literally launched things in minutes after that, and they didn't have to think about any of those underlying resources.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#43

Post by meinheld111 » Thu, Jun 08, 2023, 4:51 PM UTC

CharlesW wrote: post #27

ljm wrote: post #17

Earlier quoted context omitted.

I’ve been exploring this lately because, honestly, the cloud is total overkill for small startups and hobby projects. Kubernetes has its value even for small scale workloads like that, but it’s still a few steps more than, say, running a Capistrano script to push your code to a small Linux box with a database on a second one. You’ll get really far on minimal resources these days, especially with cheaper ARM boxes tha…

> …the cloud is total overkill for small startups and hobby projects. It absolutely can be, sure. But solutions like Vercel, Cloudflare Workers, Supabase, etc. can be excellent and inexpensive for those use cases.

And surely a vivid tech stack does more to make you look good in front of vc‘s than an overkill architecture does make you look incompetent

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#44

Post by mlhpdx » Thu, Jun 08, 2023, 4:51 PM UTC

taeric wrote: post #16

itsmemattchung wrote: post #2

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

It is rather amusing how over engineered most seed projects have a tendency to be. I do think ddb and lambda hit a sweet spot for costs on ramping up. The rest, though, really struggle.

For me, setting up connections between SQS, SNS, DDB, Lambda, step functions, S3, Route53, API Gateway in CloudFormation is just a muscle memory. I’m much faster at it at this point that I am at standing up an EC2. I agree it can be hard to learn, but it certainly isn’t hard to do.

Elsewhere in the comments, there’s a suggestion that this kind of thing isn’t appropriate for “hobby projects” and early stage but I disagree. Those are the times when you really want something you can step away from without doing a disservice to your customers (i.e. letting packages go out of date and get vulnerable) and cost you as little as possible in a steady state so you can focus on acquiring customers and not worrying about fuddling around with the guts.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#45

Post by clvx » Thu, Jun 08, 2023, 4:51 PM UTC

ljm wrote: post #17

itsmemattchung wrote: post #2

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

I’ve been exploring this lately because, honestly, the cloud is total overkill for small startups and hobby projects. Kubernetes has its value even for small scale workloads like that, but it’s still a few steps more than, say, running a Capistrano script to push your code to a small Linux box with a database on a second one. You’ll get really far on minimal resources these days, especially with cheaper ARM boxes tha…

At the beginning, just for the development experience I would just put an instance in some cloud provider and use microk8s or k3s to serve the app. It's very straightforward and then you can move to a managed service if needed. You will probably be using the same tooling and integrations at different steps. Context switching is low and you can reproduce locally. I'm down for serverless options when needed but I have a strong preference for local development.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#46

Post by frank_nitti » Thu, Jun 08, 2023, 4:52 PM UTC

KaiserPro wrote: post #11

waffletower wrote: post #9

The management of infrastructure via Terraform has a hidden engineering cost that should also be considered. Engineers can much more easily maintain, learn and introspect infrastructure via Kubernetes, despite its own complexity, given the immature, inconsistent and undeniably awkward qualities of the Terraform toolchain. Engineering time is expensive -- the morass of Terraform can easily quadruple engineering effort…

> Engineers can much more easily maintain, learn and introspect infrastructure via Kubernetes, despite its own complexity. Citation needed. K8s has a whole bunch of footguns that people who don't want to manage infra can easily blunder into. Terraform and ecs is not immature, and its fairly simple to maintain especially if they are just pushing updates without significant infra changes. (ie bumping the container vers…

ECS will always have the major disadvantage of strongly coupling your infrastructure and often your code to AWS.

They will continue to make it more appealing to lock your software into their platform than to go with their thinner facilities for OSS, doing the minimum to keep up to date with trends in open source, just enough to lure you in and create “easier” paths until you can’t afford to leave.

We have this problem with Azure - sure it’s easier to get a knucklehead to push buttons and get an app running, but after years you’ll be scrambling to reduce costs. Good luck with that when all of your terraforms use Azure Resource Manager and all of your source code uses Azure Functions. Being stuck with microsoft/amazon and a team of engineers who spent their time learning vendor-specific skills instead of the open source tech that enables it, sounds awful.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#47

Post by sieabahlpark » Thu, Jun 08, 2023, 4:54 PM UTC

intelthrow6 wrote: post #21

0xEFF wrote: post #14

Earlier quoted context omitted.

A single EC2 instance is an equally bad trade-off on the opposite side of the spectrum from over architected SQS, SNS, etc… The ideal trade off is a single Kubernetes cluster with as much in the cluster as makes sense for the team and stage of the project. As you say, toss the app on a single node to start, but the control plane is tremendously valuable from on the onset of most projects.

I don’t see the reasoning. A startup that outgrows an EC2 server will be making enough money to hire more people to scale the system properly than what was initially designed: trading away everything for development velocity. Kubernetes is not the right tool for this startup. Kubernetes is what large, old-school non-tech companies use to orchestrate resources, because it’s easier to find someone that “knows k8s” (no…

[dead]

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#48

Post by benced » Thu, Jun 08, 2023, 4:56 PM UTC

jorams wrote: post #29

danpalmer wrote: post #7

More accurately: "Given using AWS as a requirement, I recommended ECS instead of K8s". It's not really surprising that AWS's K8S setup isn't great, and their own implementation ties in more closely with other services they offer. It's lock-in. AWS provides just enough K8S to tick the box on a spec sheet, but have little incentive to go beyond that.

Since the post only mentions Kubernetes once , I don't really understand why it's in the title at all. > The team didn't have much DevOps expertise in-house, so a Kubernetes setup, even using a managed service like EKS, would have been way too complex for them at this stage, not to mention the additional costs of running the control plane which they wanted to avoid. The control plane cost makes sense, but I can't ima…

It is - EKS had fewer features.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#49

Post by oceanplexian » Thu, Jun 08, 2023, 4:56 PM UTC

ljm wrote: post #25

intelthrow6 wrote: post #21

Earlier quoted context omitted.

I don’t see the reasoning. A startup that outgrows an EC2 server will be making enough money to hire more people to scale the system properly than what was initially designed: trading away everything for development velocity. Kubernetes is not the right tool for this startup. Kubernetes is what large, old-school non-tech companies use to orchestrate resources, because it’s easier to find someone that “knows k8s” (no…

Is that right? K8S is basically another answer to Conway’s Law. Every startup I’ve worked at switched to it because then the infrastructure could map more closely to the code. Not unlike microservices at a higher level. The old-skool approach is depending on a team of SREs or sysadmins to provision hardware for you and basically handle the deployment, which K8S plus container images basically abstract away. Not to sa…

> The old-skool approach is depending on a team of SREs or sysadmins to provision hardware for you

This assumes that K8s won't require a "team of SREs". My experience is you need the same amount of SREs to maintain Kubernetes, probably more, because now you have a complicated control plane, a networking nightmare, then you layer that on top of resource-contention issues, security issues, cloud provider compatibility issues, buggy controllers, the list goes on.

The only thing K8s is great for is the maintainers, the consultants, and highly experienced SREs that inevitably have to be hired to clean up the mess that was created. This is my experience working in two similar sized environments, one with >1M containers, and another with an equivalent scale of bare metal servers.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#50

Post by danenania » Thu, Jun 08, 2023, 4:57 PM UTC

intelthrow6 wrote: post #21

0xEFF wrote: post #14

Earlier quoted context omitted.

A single EC2 instance is an equally bad trade-off on the opposite side of the spectrum from over architected SQS, SNS, etc… The ideal trade off is a single Kubernetes cluster with as much in the cluster as makes sense for the team and stage of the project. As you say, toss the app on a single node to start, but the control plane is tremendously valuable from on the onset of most projects.

I don’t see the reasoning. A startup that outgrows an EC2 server will be making enough money to hire more people to scale the system properly than what was initially designed: trading away everything for development velocity. Kubernetes is not the right tool for this startup. Kubernetes is what large, old-school non-tech companies use to orchestrate resources, because it’s easier to find someone that “knows k8s” (no…

Most startups are at least going to want to be able to deploy, scale up or down, and restart an app without downtime. I wouldn't say that's overkill.

While it's not impossible to do with a single instance, you can spend a lot of time shaving that yak. It's reasonable to pay a bit more to have that stuff handled for you in a robust way.