Earlier quoted context omitted.
And in that case, why ec2, why not a more affordable provider?
Who would you recommend as a more affordable provider?
I use digital ocean for simple projects. It’s not bad
Why I recommended ECS instead of Kubernetes to my latest customer
111–120 of 126 posts
Re: Why I recommended ECS instead of Kubernetes to my latest customer
#112Earlier quoted context omitted.
Eventually EKS was built to satisfy customers that insisted these issues were just FUD from aws to lock customers into the aws infrastructure. I mean..the customers are not wrong.
At the most charitable minimum that wasn’t the spirit of the convos internally though. These were the points of why aws didn’t think k8s was a great idea, even for customers, if they’re customers of aws rather than gcp. Aws makes money if you use them using k8s or ECS, and once you use any stateful service or spend the time to specify the eks infrastructure, you’ve got a switching cost no matter what. My thought in t…
> There is no meaningful portability between cloud providers using anything right now
Where are you getting this from? If you use k8s as base layer, lift and shift your infra or even running multi-cloud is not much harder than bringing up new region on the same cloud
Re: Why I recommended ECS instead of Kubernetes to my latest customer
#113I'm tracking cloud-hypervisor and kata containers closely. I'm convinced there is a unicorn opportunity here for the SME/private-cloud world. An easily managed cluster of lightweight, live-migratable, hardware isolated VMs running containers (as opposed to just herding containers) solves problems people actually have, as opposed to the problems k8s solves. k8s is fine for the scale of enterprise for which it is actua…
Kata container is just OCI-compatible kvm so what business need does it solve for your general acme corp that a standard docker/containerd/crio container doesn’t?
You've asked. I've taken the time to write an answer. Please read it.
Acme corp loves containers as much as everyone else. Containers provide great value. However, muddling around with docker/containerd/crio without some form of orchestration is just another path to a herd of fragile, neglected pet machines.
Acme corp is very different from the Big Tech world k8s came from. Acme corp doesn't have Linux kernel contributors and language developers and an IT payroll so large that the mundane devops people are lost in the noise. Acme corp must use what prevails and doesn't mystify. The "team" managing something is frequently one person, or less.
Acme corp ends up with a collection of pet VMs, all different. Lots of stuff is containerized. Some stuff isn't. Much of it is high-value: let one of those go down and an angry so-and-so will be on the horn right now, even if they haven't noticed for weeks. Most of it is low load: there will never ever be a world where these get reworked into scalable, stateless, distributed cloud apps.
How to get from a herd of pet VMs that happen to run containers (sometimes) to an orchestrated cluster of containers?
In my imagination the answer is something that looks like a mashup of Proxmox and docker-compose. It has the following features:
-- Orchestration: micro-VMs running containers scheduled across a cluster of nodes. The "micro-VM" term deserves some definition. I don't have a precise definition. I know Firecracker is too anemic and full featured VMs are too much. The micro-VMs of cloud-hypervisor are just about right. Above all "micro" just means simple, not necessarily small: a micro-VM that needs a lot of RAM and takes longer then 0.0003 us to start is fine.
-- Live migration: low-load, high-value applications need to stay up despite cluster node maintenance and despite never becoming candidates for re-engineering into cloud native applications. This feature is the #1 reason the VM part is necessary: live-migration is a native capability of KVM et al. that works well since forever, whereas containers (CRIU not withstanding) can't be live-migrated.
-- Trivially simple support of network transparent block storage: iSCSI and other network block storage is rampant at Acme corp because it's cheap, reliable, easy and fast enough. Re-engineering everything for dynamodb or whatever isn't an option. Fortunately, because we're running a micro-VM with its own kernel that has native support for network block (The other #1 reason for the VM part) we get this for free.
-- Simple operation: if it imposes a bunch of concepts that one can't already find in docker-compose it's wrong. Acme corp doesn't have the depth to deal with more and can't find that depth even if it wanted to, which it doesn't. Grug Brained Devops: not stupid, just instinctually uninterested in unnecessary abstraction, opaque jargon terminology, overengineering and fads.
Anyhow, that's my sincere attempt to answer your question. Respectfully, if you think you know of a solution you're likely wrong: I've wormed into every corner of that which prevails and it doesn't exist at the moment. That's why I claim there is an opportunity. I'm happy to be proven wrong, but you'd have to go a long way.
Re: Why I recommended ECS instead of Kubernetes to my latest customer
#114> When looking at the cloud resources, we noticed many On-Demand EC2 instances with relatively low CPU utilization, which can be expected considering they don't have customers yet. As a software consultant myself, I'd probably stop the conversation right there and ask why they are building such a robust distributed system — SQS, SNS, etc — without any customers. Still want to be deployed in AWS? Toss the damn app on…
Why would you plan not to have customers? Don't you think the company is able to forecast demand for a new product launch? Disney: We'd like to launch a new streaming service. Consultant: Great! You have no customers right now so you can run it on a singleton EC2 instance until you outgrow that scale! Disney: ...We expect 20 million people to sign up in the first week
> Don't you think the company is able to forecast demand for a new product launch?
I'm pretty sure "follow the forecast" is exactly what motivated that post.
In other words, the infrastructure is overkill for the initial forecast of customers.
They're not working for Disney.
Re: Why I recommended ECS instead of Kubernetes to my latest customer
#115Earlier quoted context omitted.
At the most charitable minimum that wasn’t the spirit of the convos internally though. These were the points of why aws didn’t think k8s was a great idea, even for customers, if they’re customers of aws rather than gcp. Aws makes money if you use them using k8s or ECS, and once you use any stateful service or spend the time to specify the eks infrastructure, you’ve got a switching cost no matter what. My thought in t…
> There is no meaningful portability between cloud providers using anything right now Where are you getting this from? If you use k8s as base layer, lift and shift your infra or even running multi-cloud is not much harder than bringing up new region on the same cloud
I’d refer you to the rest of what I wrote. If you have a single stack owned by a single team that has no meaningful use of the providers stateful services, yes. Otherwise, my points apply.
Re: Why I recommended ECS instead of Kubernetes to my latest customer
#116Static host stuff on an S3 bucket / static web app. Blob storage account with a table, maybe an on-demand function app. Sub $15/mo to run your thing until you get real demand, yeah. But its not new, the K8S shtick is coming from investors not tech people. And if its coming from the tech people throw them out of the door. Why are you cooking for 8000 people when 6 are coming over? Why are you building a kitchen to coo…
> Why are you cooking for 8000 people when 6 are coming over?
Place I worked at had a service running on K8s with, I think, 4 pods, and it got on average one hit every 2-3 seconds during office hours (and virtually none outside those.)
Re: Why I recommended ECS instead of Kubernetes to my latest customer
#117Earlier quoted context omitted.
This is exactly how the serverless guys "get you": Low traffic is nearly free but you pay for it on the slope of the scale cost ramp.
Sure, but aren't all cloud services notoriously expensive as you scale? At some point I assume you'd do what companies like Dropbox and Basecamp did, and re-host some or all of it.
Well, yeah, but now it starts being totally ridiculous.
For small projects the cloud is not needed, and a lot of effort that won't pay off. The only case where it'll pay off is if you "go viral" and rapidly need to increase capacity.
This is not free. While the cloud helps with scaling, your application still needs to support it. So there's a development cost to it, even when starting.
Then, if you scale, the cost makes it almost a necessity to rapidly get back off of the cloud ...
Re: Why I recommended ECS instead of Kubernetes to my latest customer
#118Earlier quoted context omitted.
> Engineers can much more easily maintain, learn and introspect infrastructure via Kubernetes, despite its own complexity. Citation needed. K8s has a whole bunch of footguns that people who don't want to manage infra can easily blunder into. Terraform and ecs is not immature, and its fairly simple to maintain especially if they are just pushing updates without significant infra changes. (ie bumping the container vers…
ECS will always have the major disadvantage of strongly coupling your infrastructure and often your code to AWS. They will continue to make it more appealing to lock your software into their platform than to go with their thinner facilities for OSS, doing the minimum to keep up to date with trends in open source, just enough to lure you in and create “easier” paths until you can’t afford to leave. We have this proble…
Look, the issue is this for a small business: you have 5 engineers, do you spend $150 a month to pay AWS to look after hosting your stuff? or do you pay an engineer >>$150 a month to create, manage and maintain a bespoke infrastructure?
Sure, you might be lucky with the engineers you have, they might be savvy enough to wrangle a couple of hosts for you. But are they backed up? what's the disaster recovery procedure like? How do you rotate keys/passcodes, how do you audit who has access?
Unless you are doing something wrong, your biggest costs are engineering time.
> disadvantage of strongly coupling your infrastructure and often your code to AWS.
you can say the same thing about any infrastructure. yes, you can migrate k8s from one physical host to another. But, for that to be effective, you need to not be using any manage services. So that means you're on the hook for all the painful things like DB state and recovery, messaging systems, etc,etc,etc.
Then you think as a business, what are you actually spending money to do? maintain the code that makes you money, or maintain the system underneath it, just in case you might need to move to save opex thats almost certainly going to be less than one engineer.
98% of companies have no issue with scale. They have issues with availability, features, backups, speed. exchanged 10% of an engineer's salary to never really have to deal with any of those issues is a good deal for most[1] companies
[1] most, but not all. However unless you are getting close to spending 1 engineer in AWS fees, moving to self hosting is nothing but premature optimisation.
Re: Why I recommended ECS instead of Kubernetes to my latest customer
#119Static host stuff on an S3 bucket / static web app. Blob storage account with a table, maybe an on-demand function app. Sub $15/mo to run your thing until you get real demand, yeah. But its not new, the K8S shtick is coming from investors not tech people. And if its coming from the tech people throw them out of the door. Why are you cooking for 8000 people when 6 are coming over? Why are you building a kitchen to coo…
> Why are you cooking for 8000 people when 6 are coming over? Place I worked at had a service running on K8s with, I think, 4 pods, and it got on average one hit every 2-3 seconds during office hours (and virtually none outside those.)
So I'd say the number of pods was appropriate.
Re: Why I recommended ECS instead of Kubernetes to my latest customer
#120we have retargeted some of our infrastructure from kubernetes and onto ecs fargate for the last 12 months and it has massively reduced errors and support tickets and also the cost
unfortunately this is a deal with the devil for vendor lock-in