Why I recommended ECS instead of Kubernetes to my latest customer • Hacker News Board

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#31

Post by politician » Thu, Jun 08, 2023, 4:40 PM UTC

itsmemattchung wrote: post #2

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

I recently spoke with some folks who declined to invest because our solution was too simple: specifically, the fact that we don't use Kubernetes was a negative signal.

That's baffling to me, but that perspective is out there too.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#32

Post by intelthrow6 » Thu, Jun 08, 2023, 4:41 PM UTC

ljm wrote: post #25

intelthrow6 wrote: post #21

Earlier quoted context omitted.

I don’t see the reasoning. A startup that outgrows an EC2 server will be making enough money to hire more people to scale the system properly than what was initially designed: trading away everything for development velocity. Kubernetes is not the right tool for this startup. Kubernetes is what large, old-school non-tech companies use to orchestrate resources, because it’s easier to find someone that “knows k8s” (no…

Is that right? K8S is basically another answer to Conway’s Law. Every startup I’ve worked at switched to it because then the infrastructure could map more closely to the code. Not unlike microservices at a higher level. The old-skool approach is depending on a team of SREs or sysadmins to provision hardware for you and basically handle the deployment, which K8S plus container images basically abstract away. Not to sa…

In my experience, giving code preferential treatment is how you end up with complexity lunacy; so I’ll add an addendum to Conway’s Law:

“Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure — and which mirrors the skills of its key creators.”

K8s is designed to solve Google problems. Your startup will not have Google problems. Your startup will have Pinterest problems, or Gitlab problems, or Reddit problems — at which point you do not need K8s; you need someone who knows infra (which I expect devs to be working on distributed systems to understand).

Using K8s in a startup context is a sign of conformist thinking, detached from any critical aspect.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#33

Post by lapser » Thu, Jun 08, 2023, 4:43 PM UTC

I find it really wild that anyone would ever recommend ECS. A developer deploying a service involves:

- Setting up certs (managed as TF) - Setting up ALBs (managed as TF) - Setting up the actual service definition (often done as a JSON, that is passed into TF)

Possibly other things I'm forgetting.

Some other things. It requires a *developer* to know about certs and ALBs and whatever else.

With EKS, this can all be automated. The devops engineer can set it up so that deploying a service automatically sets up certs, LBs etc. Why are we removing such good abstractions for a proprietary system that is *supposed* to be less management overheads, when in reality, it causes devs to do so much more, and understand so much more?

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#34

Post by lijok » Thu, Jun 08, 2023, 4:44 PM UTC

waffletower wrote: post #9

The management of infrastructure via Terraform has a hidden engineering cost that should also be considered. Engineers can much more easily maintain, learn and introspect infrastructure via Kubernetes, despite its own complexity, given the immature, inconsistent and undeniably awkward qualities of the Terraform toolchain. Engineering time is expensive -- the morass of Terraform can easily quadruple engineering effort…

> Engineers can much more easily maintain, learn and introspect infrastructure via Kubernetes

Hahahaha

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#35

Post by lapser » Thu, Jun 08, 2023, 4:44 PM UTC

Torwald wrote: post #28

pmarreck wrote: post #24

The article never mentions what ECS stands for, that’s not yet a TLA I’m familiar with

It stands for Enhanced Chip Set as improvement from OCS, which stands for Original Chip Set. Note that even ECS found it's successor AGA, which stands for Advanced Graphics Architecture. AGA can display 256 colors even without using HAM (hold and modify).

Assuming you're joking, ECS here stands up Elastic Container Service.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#36

Post by honkycat » Thu, Jun 08, 2023, 4:44 PM UTC

I was not convinced by this article that ECS was the right choice. It felt more like a contrarian choice.

> ECS is also relatively simple and not so far from their Docker-compose setup, but much more flexible and scalable. It also enables us to convert their somewhat stateful pets to identically looking stateless cattle that could be converted to Spot instances later.

Have you ever built something in ECS? I have, and it is missing HUGE SWATHS of the convenient functionality that EKS provides. It lacks the network effect of being a widely-used product, so searching for issues is a constant issue. It breaks and nobody knows how to help.

"Not far from their docker-compose setup..." What are you even talking about? ECS is massively more complex than docker-compose and the main similarity I see between them is that they both run docker. It's similar to docker-compose if you ignore the fact that you need permissions, load balancers, networking, etc. Which is the hard part, NOT running some containers on EC2, by the way.

It has it's own bizarre and verbose container deployment spec that is less portable, less flexible, less feature-ful, and less widely used than EKS.

> ECS will also offer ECS container logs and metrics out of the box, giving us better visibility into the application and enabling us to right-size each service based on its actual resource consumption, in the end allowing us to reduce the number of instances in the ECS cluster once everything is optimized.

Something you also get with EKS. So half of the reasons you have claimed ECS was the right choice are now in the garbage.

What you DON'T get with ECS is awesome working-out-of-the-box open source software like External Secrets, External DNS, LetsEncrypt, the Amazon Ingress Controller, argo rollouts, services, ingresses, cronjobs... I could go on and on.

They are going to try and hire DevOps engineers, and they will all have to ramp up ( and likely complain about ) ECS instead of having people walk on already prepared and ready to start implementing high quality software on a system they already know.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#37

Post by SparkyMcUnicorn » Thu, Jun 08, 2023, 4:45 PM UTC

mywittyname wrote: post #15

waffletower wrote: post #10

Earlier quoted context omitted.

And in that case, why ec2, why not a more affordable provider?

There's a lot of expertise in AWS-land.

A DevOps junior should be able to start a VM just about anywhere, and without specialized experience.

AWS/GCP/Azure knowledge definitely helps when deploying there, but it's also not really necessary to get something running.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#38

Post by honkycat » Thu, Jun 08, 2023, 4:45 PM UTC

lapser wrote: post #33

I find it really wild that anyone would ever recommend ECS. A developer deploying a service involves: - Setting up certs (managed as TF) - Setting up ALBs (managed as TF) - Setting up the actual service definition (often done as a JSON, that is passed into TF) Possibly other things I'm forgetting. Some other things. It requires a *developer* to know about certs and ALBs and whatever else. With EKS, this can all be au…

Agreed. After recently finishing up a migration off ECS, it is madness and feels like OP just wanted a contrarian take.

Honestly, if they had said: "So instead we set up some bare-metal EC2 instances" I would be on-board.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#39

Post by NovemberWhiskey » Thu, Jun 08, 2023, 4:47 PM UTC

lapser wrote: post #33

I find it really wild that anyone would ever recommend ECS. A developer deploying a service involves: - Setting up certs (managed as TF) - Setting up ALBs (managed as TF) - Setting up the actual service definition (often done as a JSON, that is passed into TF) Possibly other things I'm forgetting. Some other things. It requires a *developer* to know about certs and ALBs and whatever else. With EKS, this can all be au…

That seems a strange way to look at things to me. If you're going to give credit for things that a devops engineer can do inside the Kubernetes platform, why not given equivalent credit for what a devops engineer can do with a Terraform module that would achieve substantially similar levels of automation and integration with ECS?

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#40

Post by politician » Thu, Jun 08, 2023, 4:47 PM UTC

ljm wrote: post #25

intelthrow6 wrote: post #21

Earlier quoted context omitted.

I don’t see the reasoning. A startup that outgrows an EC2 server will be making enough money to hire more people to scale the system properly than what was initially designed: trading away everything for development velocity. Kubernetes is not the right tool for this startup. Kubernetes is what large, old-school non-tech companies use to orchestrate resources, because it’s easier to find someone that “knows k8s” (no…

Is that right? K8S is basically another answer to Conway’s Law. Every startup I’ve worked at switched to it because then the infrastructure could map more closely to the code. Not unlike microservices at a higher level. The old-skool approach is depending on a team of SREs or sysadmins to provision hardware for you and basically handle the deployment, which K8S plus container images basically abstract away. Not to sa…

Conway's law is about mapping teams to code+infrastructure (generally: areas of responsibility), not about mapping code to infrastructure. It's about people and politics.

You're right that K8S is an answer to Conway's Law: our people don't get along or can't collaborate or we have too many of them, so we will split them into team per service and force them to collaborate over network interfaces. Likewise, the infrastructure people will communicate with the other teams using Dockerfiles.