Live data from Hacker News

Why I recommended ECS instead of Kubernetes to my latest customer

leanercloud.beehiiv.com

51–60 of 126 posts

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#51

More accurately: "Given using AWS as a requirement, I recommended ECS instead of K8s". It's not really surprising that AWS's K8S setup isn't great, and their own implementation ties in more closely with other services they offer. It's lock-in. AWS provides just enough K8S to tick the box on a spec sheet, but have little incentive to go beyond that.

As someone who sat in on the product development discussions at aws about EKS the internal view was K8S was: * a lock in strategy by Google to substitute for the fact they don’t yet have systemic abstractions at a provider level. By “owning” the design and engineering around k8s through capture they can ensure the backing services they build in gcp naturally support k8s users as they develop their roadmap * the providing of customer space SDN and infrastructure services via a OS/user space runtime was seriously weaker than what an infrastructure provider can offer in terms of stability, durability, security, audit, etc. * the complexity of running an abstraction layer on top of an abstraction layer that provide essentially identical or similar services was crazy * the semantics of durable stores (queues, databases, object stores, etc) would never be sufficient in a k8s model compared to a hosted provider service * the bridging of k8s into provider durable stores breaks the run anywhere model as even stores like s3 that have similar APIs across providers have vastly different semantic behaviors * as such, k8s solved mostly stateless problems, which being trivial doesn’t merit the complexity of k8s * k8s wasn’t a standard, that requires standardization and a standards body. K8s was a popular solution to a problem that has had many solutions. Google promoting it and investing in it didn’t make it a standard, nor did the passion of the k8s community. * that said for customers in data center installs would benefit from the software defined infrastructure and isolation as most data center installs are giant undifferentiated flat prod blobs of badness. The same could be said for all the various similar solutions to k8s though and it wasn’t obvious why k8s was the “right” choice beyond the hype cycle at that time.

Eventually EKS was built to satisfy customers that insisted these issues were just FUD from aws to lock customers into the aws infrastructure. However what I have seen since is a basic progression of: customer uses k8s on prem, is fanatical about its use. They try to use it in aws and it’s about as successful as on prem. Their peers squint at it and say “but wouldn’t this be easier with ECS/fargate?” K8s folks lose their influence and a migration happens to ECS. I’ve seen this happen inside aws working with customers and in three megacorps I’ve worked on cloud strategies for. I’ve yet to encounter a counter example, and this was sort of what Andy predicted at the time. I’m not saying there aren’t counter examples, or that this isn’t a conspiracy against k8s to get your dollars locked into aws.

On standards Andy always said that at some point cloud stuff would converge into a standards process but at the moment too little is known about patterns that work for standards to be practical. Any company launching into standards this early would get bogged down and open the door to their competitors innovating around them and setting the future standard once the time is right for it. Obviously not an unbiased viewpoint, but a view that’s fairly canonical at Amazon.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#52

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

>ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers

I think this is one of those things that really depends on the use case. If they are performing expensive inference, I think having any queue is better than no queue. Going from a synchronous system to an asynchronous one is not easy and it's not something you would want anyone to be paged for once it starts to matter. Getting SQS/SNS up and running now could be a couple hours of work today and is practically free if your traffic is low.

Similarly I have a number of side projects that run extremely cheaply just using ECS and Fargate. I don't even think about Kubernetes really, it's just a PaaS to me that I'm shipping ARM binaries to. As a result I don't think very hard about autoscaling, failover, load balancing or deployment. A github action just pushes master to ec2 and everything "just works".

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#53
post #44
post #16

Earlier quoted context omitted.

It is rather amusing how over engineered most seed projects have a tendency to be. I do think ddb and lambda hit a sweet spot for costs on ramping up. The rest, though, really struggle.

For me, setting up connections between SQS, SNS, DDB, Lambda, step functions, S3, Route53, API Gateway in CloudFormation is just a muscle memory. I’m much faster at it at this point that I am at standing up an EC2. I agree it can be hard to learn, but it certainly isn’t hard to do. Elsewhere in the comments, there’s a suggestion that this kind of thing isn’t appropriate for “hobby projects” and early stage but I disa…

Your muscles must be tuned to enormous amounts of IAM-fu ;-)

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#54

Earlier quoted context omitted.

And in that case, why ec2, why not a more affordable provider?

There's a lot of expertise in AWS-land.

There are a lot of salesforce consultants but still would advise to look at other solutions.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#55

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

And in that case, why ec2, why not a more affordable provider?

Because I already have an AWS account that bills directly to my credit card along with some other stuff that I'm already paying for. Every time I go down the let me save money route I spend hours reading through CD website reviews for hosting providers without any real understanding of their quality to save a few dollars and end up burning tens of hours of time. Or I could just fire the fucking thing up on AWS and then turn it off if I decide not to work on the project further

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#56
Mostly a note to self: it is interesting to read this account and connect it to the financial planning case studies that show up in personal finance blogs and articles. It seems like there’s a lot of shared terminology and practice between the domains.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#57

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

With so many developers I feel that there is a complete lack of familiarity with what it takes to just run a website. So many came up in the land of cloud and k8s and etc. There are use cases for these more advanced production environments. But if more developers just learned how to make a website on linux, with a db, a webserver, and an application. They would know that a lot of more complex things just aren't neede…

Truly, a very small number of real servers, just enough for blue/green deployments and so you can stay up if any one server goes offline, meets any plausible needs for a really, really high percentage of businesses & products. A ton of early-stage ones can get away with skipping most of that and just run on one or two servers, period, for quite a while.

If you're outsourcing operations to AWS or whomever, a couple largish instances and a couple supporting services can get you pretty much that same thing, for a bit more money and a bit less control over performance-consistency.

All that HA/scaling/clustering/cloud stuff is expensive, not just in monetary terms, but in performance terms. If you don't actually need it, a high percentage of your compute & (especially) your network traffic may be going to that, rather than actually serving the product. It also adds a hell of a lot of complexity, which comes at a significant time-cost for development, unless you want your defect rate to shoot up.

> But if more developers just learned how to make a website on linux, with a db, a webserver, and an application.

And hell, nothing's stopping you from writing 12-factor apps and deploying containers, and scripting your server set-up and config, even if you don't go straight for heavy, "scalable" architecture. Even if your server's a beige Linux box in a closet. Enough benefits that the effort's probably a wash at worst (hey, documentation you can execute is the best documentation!) even if you never need to switch architectures, and then you'll have a relatively easy time of it, if you do end up needing to.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#58
IMO ECS is in a weird position right now, because:

1 - It's simpler thank K8s, but not that much simpler than your avg managed K8s offering

2 - It really locks you in the AWS ecosystem

3 - It is way less used than K8s or just running things on servers, so there are way less help / learning resources

I really don't see how using ECS is much better than EC2 + compose for small setups and this post didn't provide many good arguments to convince me.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#59
I'm tracking cloud-hypervisor and kata containers closely. I'm convinced there is a unicorn opportunity here for the SME/private-cloud world. An easily managed cluster of lightweight, live-migratable, hardware isolated VMs running containers (as opposed to just herding containers) solves problems people actually have, as opposed to the problems k8s solves. k8s is fine for the scale of enterprise for which it is actually intended and the problem space it was designed to address. It's not fine for everything else.

Re: Why I recommended ECS instead of Kubernetes to my latest customer

#60

> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…

Why would you plan not to have customers? Don't you think the company is able to forecast demand for a new product launch?

Disney: We'd like to launch a new streaming service.

Consultant: Great! You have no customers right now so you can run it on a singleton EC2 instance until you outgrow that scale!

Disney: ...We expect 20 million people to sign up in the first week

Post reply on HN