81–90 of 216 posts
As of now, it seems like my only alternative is to run k8s on a 2GB of RAM system, so I'm considering moving to Hetzner just to run k3s or k0s.
How to save 1M off your cloud infra? Start from a 2M bill. That's how I see most of these projects. You create a massively expensive infra because webscale, then 3 years down the road you (or someone else) gets to rebuild it 10x cheaper. You get to write two blog posts, one for using $tech and one for migrating off $tech. A line in the cv and a promotion. But kudos for them for managing to stop the snowball and actua…
It makes sense to use whatever $tech helps you get an MVP out asap and iterate on it. once you're sure you found gold, then it makes sense to optimize for scale. The only thing I guess one has to worry about when developing something like that, is to make sure good scalability is possible with some tinkering and efforts and not totally impossible.
Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…
Then we have test environments, around 20 of them: 20 clusters.
Then there are 10s of clusters installed on customers' infra.
So 47 clusters isn't really a huge/strange number.
Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…
There are genuine reasons for running multiple clusters. It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation so they get their own cluster (although somehow its ok that the nodes are still VMs that are probably running on shared nodes… these requirements can be arbitrary sometimes).
Earlier quoted context omitted.
We ran only two (very small) clusters for some time in the past and even then it introduced some unnecessary overhead on the ops side and some headaches on the dev side. Maybe they were just growing pains, but if I have to run Kubernetes again I will definitely opt for a single large cluster. After all Kubernetes provides all the primitives you need to enforce separation. You wouldn't create separate VMWare productio…
You need a separate cluster for production because there are operations you'd do your staging/QA environments that might accidentally knock out your cluster, I did that once and it was not fun. I completely agree with keeping everything as simple as possible though. No extra clusters if not absolutely necessary, and also no extra namespaces if not absolutely necessary. The thing with Kubernetes is that it was designe…
I agree that k8s offers many features that most users probably don't need and may not even know of. I found that I liked k8s best when we used only a few, stable features (only daemonsets and deployments for workloads, no statefulsets) and simple helm charts. Although we could have probably ditched helm altogether.
Earlier quoted context omitted.
Kubernetes upgrades? Don't those risk bricking everything with just 1 environment?
Why would those brick everything? You update node one by one and take it slow, so issues will become apparent after upgrade and you have time to solve those - whole point of having clusters comprised of many redundand nodes.
Edit: And even when you "brick" it and cannot roll back, there is still a way to bring your control plane back by using an etcd backup, right?
Why exactly did they have 47 clusters? One thing I noticed (maybe because I’m not at that scale) is that companies are running 1+ clusters per application. Isn’t the point of kubernetes that you can run your entire infra in a single cluster, and at most you’d need a second cluster for redundancy, and you can spread nodes across regions and AZs and even clouds? I think the bottleneck is networking and how much crossta…
What else is out there? I'm running docker swarm and it's extremely hard to make it work with ipv6. I'm running my software on a 1GB RAM cloud instance and I pay 4EUR/month, and k8s requires at least 1GB of RAM. As of now, it seems like my only alternative is to run k8s on a 2GB of RAM system, so I'm considering moving to Hetzner just to run k3s or k0s.
ie nginx has the public facing ipv4 & ipv6 address(es), with docker swarm behind that communicating to the nginx proxies over ipv4
Earlier quoted context omitted.
> "Create an account to read the full story." Why is this required?
They are also misleading the users with that message. I signed up, but now I got a message that I have to upgrade to a paid account... If they had told me that from the start, I would not bother with creating an account at all. The message implies that a free account would be enough
Then I read someone saying that K8s should never be used for teams Our team is smaller than 20FTE, so we use AWS/Fargate now. Works like a charm.