Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I recommended ECS instead of Kubernetes to my latest customer

leanercloud.beehiiv.com

Post reply on HN
Print view View original

101–110 of 126 posts

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#101

Post by alien_ »

jpgvm wrote: post #70
tedivm wrote: post #42

Earlier quoted context omitted.

I honestly don't understand where you're coming from. If a devops engineer can set things up on eks for people to launch without thinking of those things, what's stopping that same engineer from doing similar for ecs? When I was at Rad AI we went with ECS. I made a terraform module that handled literally everything you're talking about, and developers were able to use that to launch to ECS without even having to thin…

Handing Terraform to developers has it's own host of issues. A major benefit of k8s that is usually massively overlooked is it's RBAC system and specifically how nice a namespace per team or per service model can be. It's probably not something a lot of people think about until they need to handle compliance and controls for SOC II and friends but as someone that has done many such audits it's always been great to be…

The OP here.

What you say may make sense for a large corporation with hundreds of developers from many teams, all sharing a single cluster, but remember this is a pre-revenue startup with a single dev team of less than a dozen people.

But then with a large cluster you will struggle with splitting the costs. In such scenarios I'd rather give each team its own AWS account and have some devops people set up everything from the landing zone.

In this particular case, every service is set up from less than 100 lines of Terraform, which includes Docker image build and push, as well as the task and service definition that deploys that docker image.

yes, they need to handle Terraform, but it's really not so different from the previous Docker-compose YAML file, not to mention the way it would look if converted to K8s YAML.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#102

Post by rdsubhas »

What they didn't appear to have considered – was the Dev side of DevOps. Kubernetes runs on developer machines and single-node CI agents. In my company, all CI agents are single-node k3s clusters, all our engineers kubectl apply their services there for integration and e2e testing, same environment from dev to prod. We provide the same single-node VMs for development on the cloud, and Podman desktop for local kubernetes. It has hooks to inject stuff (injecting centralized secrets, configuration, sidecars, etc) in a single way, no need to implement centralized features separately for CI and separately for Prod. It has hooks to validate & reject stuff that doesn't comply with org policies (e.g. limit only core workloads, upper bounds on cpu/memory, volumes, validate everyone sticks to core workload specs and do not use any alpha/beta APIs, etc) so that SRE can allow decentralization while still being in control of what runs and how.

ECS is a deployment tool. Kubernetes is a dev-to-ci-to-prod tool, providing same environment for standard workload specs across the full development cycle, and a single way to inject common features into the standard workloads.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#103

Post by mejutoco »

djbusby wrote: post #79
meinheld111 wrote: post #43

Earlier quoted context omitted.

And surely a vivid tech stack does more to make you look good in front of vc‘s than an overkill architecture does make you look incompetent

IME the investor cares more about Traction that Technology.

IDK, I remember seeing a tweet from Paul Graham saying that any new startup should use Typescript (I guess instead of js) so there might be some rules of thumb that some investors follow.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#104

Post by alien_ »

honkycat wrote: post #38
lapser wrote: post #33

I find it really wild that anyone would ever recommend ECS. A developer deploying a service involves: - Setting up certs (managed as TF) - Setting up ALBs (managed as TF) - Setting up the actual service definition (often done as a JSON, that is passed into TF) Possibly other things I'm forgetting. Some other things. It requires a *developer* to know about certs and ALBs and whatever else. With EKS, this can all be au…

Agreed. After recently finishing up a migration off ECS, it is madness and feels like OP just wanted a contrarian take. Honestly, if they had said: "So instead we set up some bare-metal EC2 instances" I would be on-board.

The OP here, thanks for the comment!

It was definitely not about being contrarian but about offering first and foremost a more cost effective but still relatively simple, scalable and robust alternative to their current setup.

They have a single small team of less than a dozen people, all working on a single application, with a single frontend component.

Imagine instead this team managing a K8s setup with DNS, ALB and SSL controllers that each set up a single resource. I personally find that overkill.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#105

Post by tedivm »

jpgvm wrote: post #70
tedivm wrote: post #42

Earlier quoted context omitted.

I honestly don't understand where you're coming from. If a devops engineer can set things up on eks for people to launch without thinking of those things, what's stopping that same engineer from doing similar for ecs? When I was at Rad AI we went with ECS. I made a terraform module that handled literally everything you're talking about, and developers were able to use that to launch to ECS without even having to thin…

Handing Terraform to developers has it's own host of issues. A major benefit of k8s that is usually massively overlooked is it's RBAC system and specifically how nice a namespace per team or per service model can be. It's probably not something a lot of people think about until they need to handle compliance and controls for SOC II and friends but as someone that has done many such audits it's always been great to be…

Huh, I've had no issues with AWS IAM or Auditors when it comes to Terraform. I've managed compliance at multiple startups too.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#106

Post by ljm »

CharlesW wrote: post #27
ljm wrote: post #17

Earlier quoted context omitted.

I’ve been exploring this lately because, honestly, the cloud is total overkill for small startups and hobby projects. Kubernetes has its value even for small scale workloads like that, but it’s still a few steps more than, say, running a Capistrano script to push your code to a small Linux box with a database on a second one. You’ll get really far on minimal resources these days, especially with cheaper ARM boxes tha…

> …the cloud is total overkill for small startups and hobby projects. It absolutely can be, sure. But solutions like Vercel, Cloudflare Workers, Supabase, etc. can be excellent and inexpensive for those use cases.

I consider them separate from the cloud on the basis they’re offering a platform as a service that just happens to re-sell cloud resources.

If you tried to replicate them on the same cloud provider, end to end, it would cost far more than they charge.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#107

Post by CharlesW »

mikeravkine wrote: post #96
CharlesW wrote: post #92

Earlier quoted context omitted.

So far, for my hobby projects it's basically free.

This is exactly how the serverless guys "get you": Low traffic is nearly free but you pay for it on the slope of the scale cost ramp.

Sure, but aren't all cloud services notoriously expensive as you scale? At some point I assume you'd do what companies like Dropbox and Basecamp did, and re-host some or all of it.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#108

Post by lapser »

alien_ wrote: post #98
lapser wrote: post #33

I find it really wild that anyone would ever recommend ECS. A developer deploying a service involves: - Setting up certs (managed as TF) - Setting up ALBs (managed as TF) - Setting up the actual service definition (often done as a JSON, that is passed into TF) Possibly other things I'm forgetting. Some other things. It requires a *developer* to know about certs and ALBs and whatever else. With EKS, this can all be au…

The OP here, thanks for the comment! Why does the developer need to care about the certs and ALBs? The devops engineer you need to set up all those controllers could as well deploy those resources from Terraform. As I showed in the diagrams from the article this application has a single ALB and a single cert per environment and the internal services only talk to each other through the rabbit MQ queue. DNS, ALB and TL…

> DNS, ALB and TLS certs could be easily handled from just a few lines of Terraform, and nobody needs to touch it ever again.

Welcome to reality, where this is not the case.

I'm currently working at a company where we're using TF and ECS, and app specific infra is supposedly owned by the service developers.

In reality, what happens is devs write up some janky terraform, potentially using the modules we provide, and then when something goes wrong, they come to us cos they accidentally messed around with the state or whatever. DNS records change. ALB listener rules need to change.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#109

Post by pnpnp »

interroboink wrote: post #63
sigstoat wrote: post #13

Earlier quoted context omitted.

i had a client who was burning… $10k? maybe $20k per month largely on nodes for EKS when they had no paying customers and ~zero load. (they had fully “production” sized clusters in all of their environments, and they had a slew of weird not-quite-prod environments.) they also had some rabbitmq-on-k8s system going that fell over during small tests because they couldn’t get k8s to actually scale it. (which then convinc…

The promise of cloud infrastructure is that it can scale to fit demand — start small, and grow as needed. But sometimes the truth is that it just lets people spend money more easily (: Back in the day, it would have required a whole procedure to buy that hardware, have it set up, etc. Now you can needlessly spend $10k per month with just a few clicks!

This is one reason I like serverless. It works for a bunch of cases when you can wrap your head around it, and cost can scale linearly with your growth.

At some point, it might make sense to move off for cost reductions, but tools like GCP Cloudrun (deploy dockerized app servers that scale dramatically better than k8s) can be really nice for a small team.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#110

Post by beaviskhan »

jpgvm wrote: post #70
tedivm wrote: post #42

Earlier quoted context omitted.

I honestly don't understand where you're coming from. If a devops engineer can set things up on eks for people to launch without thinking of those things, what's stopping that same engineer from doing similar for ecs? When I was at Rad AI we went with ECS. I made a terraform module that handled literally everything you're talking about, and developers were able to use that to launch to ECS without even having to thin…

Handing Terraform to developers has it's own host of issues. A major benefit of k8s that is usually massively overlooked is it's RBAC system and specifically how nice a namespace per team or per service model can be. It's probably not something a lot of people think about until they need to handle compliance and controls for SOC II and friends but as someone that has done many such audits it's always been great to be…

Make Terraform run only off git repos, and control commit rights to that repo. That's been a successful approach for me in the past when dealing with auditors.
Post reply on HN