Live data from Hacker News

Kubernetes is a red flag signalling premature optimisation

jeremybrown.tech

401–410 of 558 posts

Re: Kubernetes is a red flag signalling premature optimisation

#401

The problem with this sort of argument is that it sets up a straw-man, without describing what you should do instead. Kubernetes could be better - of course - but what better approach is the author recommending to use instead? It would be better if we could use one language on frontend and backend - of course - but what is this one language that works everywhere? It would be better - of course - if there was a SaaS t…

> The problem with this sort of argument is that it sets up a straw-man, without describing what you should do instead. Kubernetes could be better - of course - but what better approach is the author recommending to use instead? The article explicitly advocates for high-level PaaS tools as an alternative to Kubernetes.

Maybe it could be read that way, but I think it's a good indication of the problem that a strict reading doesn't include that. The article advocates that you should be using Heroku or Vercel or Netlify or Fly as not doing so is an antipattern, but it then says that instead of k8s "Most organisations should consider some of the higher-level building blocks available via cloud providers", which seems contradictory.

I quite liked the actual message of the article (which I take as "be mindful of your technology choices") - but I think that it is really devalued by the less coherent but more click-baity arguments (e.g. "javascript is the only rational language choice")

Edit: Actually, re-reading the article yet again, I think I can see your reading (though I wish you had written the article, as your writing is much clearer). As I understand it, the article says you should use a PaaS-style solution until you outgrow it, when you should move to something from the cloud providers. For me, that would be CloudRun -> GKE-Autopilot -> GKE (skipping CloudRun if you don't want to learn two systems), but there may be some google-bias there!

Re: Kubernetes is a red flag signalling premature optimisation

#402
post #278

I agree entirely. I like to call what the author is referring to as, "What-If Engineering". It's the science of thinking you'll be Google next week, so you build for that level of scale today. It involves picking extremely complicated, expensive (both in compute and skilled labour) technologies to deploy a Rails app that has two features. And it all boils down to, "But what if..." pre-optimising. It happens at all le…

Sorry but managed k8s is really simple and wildly a better pattern than just running VMs. You don’t need google scale for it to help you, and spinning things up without understanding the maintenance cost is just bad engineering

You want simple? Heroku, Render, DigitalOcean AppPlatform

Re: Kubernetes is a red flag signalling premature optimisation

#403

I'm getting tired of the "You don't actually need Kubernetes while starting out!" crowd, despite being part of it. Of course you don't. Of course if you don't know Kubernetes, learning it as you try to get a company going on a minimum headcount is not the most efficient approach. But for pete's sakes man, if you have used K8s before, know what you're doing and you're running on cloud, just shove a couple off-the-shel…

I think it still bears repeating.. I've seen projects fail or take 10x as long due to choosing to go with k8s day 1 instead of starting with even a basic VM. It may be easy enough to get k8s itself setup. However, without an in-house expert sat within the dev team, all the access/permissions to resources to/from the cluster were hell. NFS disk Read/Write, S3 Read/Write/List, RDS DB access, Kafka access, corp network…

It’s not like vms have magical simplicity guardrails. I run all my personal stuff in K8s because it’s what I know and it’s all pretty set it and forget it. If it really is that simple, or your comfortable with Ansible great! but I’ve seen lots of issues on VMs with mountains of custom deploy scripts, or OS’ that can’t be upgraded, or manual configurations people don’t understand.

Re: Kubernetes is a red flag signalling premature optimisation

#404
post #206

Earlier quoted context omitted.

It’s really complicated when something goes wrong. That is my only criticism. Particularly in the various CNI layers out there. You really have to know exactly how everything works to get through those days and that is beyond the average person who can create a docker container and push it into the cluster which is the usual success metric.

In what cases would I have to worry about that?

If it breaks. If you have an outage because of weird K8s networking issues (and I've seen them) you'll suddenly care very much.

Re: Kubernetes is a red flag signalling premature optimisation

#405
I remember working with a client in the last 5 years that demanded a Kubernetes cluster to run custom analytics on very fast incoming data streams (several GB per hour). By "custom analytics" I mean, python scripts that loaded a day's worth of data, computed something, wrote the results to disk, and quit.

During development of the scripts, the developers/data scientists wrote and tested everything outside of the cluster since they were simple scripts. They had no problem grinding through a day's worth of data in their testing. But going into prod, had to shove it into the cluster. So now we had to maintain the scripts AND the fscking cluster.

Why?

"What if the data volume increases or we need to run lots of analytics at once?"

"You'll still be dominated by I/O overhead and your expected rate of growth in data volume is Nope, had to have the cluster. So we had the cluster. At the expense of 10x the hardware, another rack of equipment, a networking guy, and a dedicated cluster admin (with associated service contracts from a support vendor). It literally all ran fine on a single system with lots of RAM and SSDs -- which we proved by replicating all of the tasks the cluster was doing.

Argh...

Re: Kubernetes is a red flag signalling premature optimisation

#406

Earlier quoted context omitted.

I don't think their advice about not using it in a startup is correct either. You just need to somewhat know what you're doing. I know of such a case, where a single engineer could leverage the helm chart open source community, and set up a scalable infrastructure, with prometheus, grafana, worker nodes that can scale independently of web service, a CI/CD pipeline that can spin up complete stacks with TLS automated t…

> with prometheus, grafana, worker nodes that can scale independently of web service, a CI/CD pipeline that can spin up complete stacks with TLS automated through nginx and cert-manager, do full integration tests, etc. > I found that to be quite impressive, for one person, one year, and would probably be completely impossible if it wasn't for k8s. I've always found this interesting about web based development. I have…

> Prometheus, grafana

In short. Prometheus is a worker that knows about all your other services. Each service of interest can expose an endpoint that prometheus scrapes periodically. So the services just say what the current state is, and prometheus, since it keeps asking, knows and stores what happens over time. Grafana is a web service that uses prometheus as a data source and can visualize it very nicely.

Prometheus also comes with an Alert Manager, where you can set up rules for when to trigger an alert, that can end up as an email or slack integration.

They are all very useful, and gives a much needed insight into how things are going.

Re: Kubernetes is a red flag signalling premature optimisation

#407

Earlier quoted context omitted.

The thing is, unless using those technologies was somehow core to what the single engineer was trying to, it might be technically impressive but might not have actually provided value for users. Users don't really care if you have a really impressive stack with cool technologies if it doesn't offer anything more than a couple of web servers and a DB server.

Right on. Previous devs at co I joined wanted to play DevOps cowboys. They used Ansible scripts to spin up various AWS services costing the company over 100K/yr. New lead came in, got rid of that crap by using 3rd party services to spin up infrastructure. Got a load balancer, a few VMs + DB. Reduced the cost down by 85% and greatly simplified the entire stack. I learned a really valuable lesson without having to make…

I know this utterly misses the larger point of your comment, but:

> They used Ansible scripts to spin up various AWS services

This seems less about using the "cool/new" tech... rather it's about using the "right" tech. Config management tools like Ansible/Chef/Puppet are very much previous-generation when it comes to cloud infrastructure.

They... can manage cloud infrastructure, but they were created prior to the ubiquity of cloud deployments, and the features are glued on. Not choosing a more modern IaC framework tells me they(those devs) were going to be making sub-optimal implementation decisions regardless.

Re: Kubernetes is a red flag signalling premature optimisation

#408
post #171

Earlier quoted context omitted.

I mean, container+registry is possibly more complex to start and maintain than just creating a service file with SystemD. Deployment can be as easy of "systemctl restart service", and seems to me that configuring all of those resources in Kubernetes is far more difficult that just setting up a simple service on a bare box. Not to mention that you could use Docker too. And by the looks of it, ingress doesn't seem triv…

You don't run your own registry (except in very rare circumstances where that makes sense). You author Dockerfile (or better yet use a tool like jib that creates containers automatically without even a Docker daemon), then you push to hosted registry. Ingress is trivial to use. Internally it's less trivial but you don't need to peek inside unless you manage to break it which generally speaking, you won't; even if you…

> You don't run your own registry (except in very rare circumstances where that makes sense). You author Dockerfile (or better yet use a tool like jib that creates containers automatically without even a Docker daemon), then you push to hosted registry.

Dockerfiles aren't trivial. I've helped migrate some services to docker and it's not "just put it in Docker and that's it". Specially because most applications aren't perfectly isolated.

> Ingress is trivial to use. Internally it's less trivial but you don't need to peek inside unless you manage to break it which generally speaking, you won't; even if you think whatever you are doing is very special it probably does it already. That is the benefit of literal thousands of teams using the exact same abstraction layer.

Literal thousands of teams also use Nginx and still doesn't mean that there aren't configuration errors, issues and other things that are to debug. Not to mention that a lot of applications can run without requiring a reverse proxy.

> You don't have SSH (this is objectively good), instead you can get exec on a container if you need it (and that container contains a shell). You can now control on a fine grained and integrated fashion exactly who is allowed to exec into a container and you get k8s audit events for free. (and if you are using a hosted system like GKE then it automatically flows into said providers audit system).

And that's cool if you need that, but "ssh user@machine" looks far easier.

> The point is you can buy managed k8s, there isn't an equivalent for old school sysadmin that doesn't amount to outsourcing to a body shop.

Look, the other day I had to set up a simple machine to monitor some hosts in a network. I've Ansible to automate it, but ultimately it boils down to "sudo apt install grafana prometheus; scp provisioning-dashboards; scp prometheus-host-config; scp unattended-upgrades-config" plus some hardening configs. I don't need redundancy, uptime is good enough with that. If Prometheus can't connect to the hosts, I can test from the machine, capture traffic or whatever and know that there isn't anything in the middle. If Grafana doesn't respond I don't need to worry whether the ingress controller is working well or if I missed some configuration.

Could I run that with Kubernetes? Well, managed k8s is already out of the window because the service is internal to an enterprise network. And self-hosted kubernetes? I still need to do the same base linux box configuration, plus setting up k8s, plus setting up the infra and networking with k8s plus the tools I actually want to use.

And that's the point. I am not trying to identify any k8s downsides, what I said that setting up k8s looks far more confusing than setting up a standard Linux box, and by the things you say I am even more convinced than before than it is indeed. I still need to understand and run the programs I want (which is most of the cost of deployment), but I also need to understand Kubernetes and, if I am using a managed platform, I need to understand how the managed platform works, configure it, link it with my system... In other words, not only do I need the general knowledge for application deployment but also the specific implementation in Kubernetes.

Re: Kubernetes is a red flag signalling premature optimisation

#409

Earlier quoted context omitted.

Not OP, but App Engine have actually been around for a long time. Also, there's no inherent lock-in, you can basically just deploy it somewhere else. Data is where the lock-in lies. Moving can be hard if you use proprietary databases. Can still be worth it.

>App Engine have actually been around for a long time That means very little, I hope you realize. Reader, Voice, Chat, etc.[0] were all around a long time. >Also, there's no inherent lock-in GAE has plenty of proprietary APIs you can depend on. Whether or not you do is up to the programmer. 0 - A comment below notes that voice and chat aren't deprecated yet. Voice was announced deprecated, and google has had so many…

Gae is nearly twice as old as reader was when deprecated. Voice and chat still exist and aren't deprecated.

Re: Kubernetes is a red flag signalling premature optimisation

#410

Earlier quoted context omitted.

Are there contractors out there that will take on call shifts? Because it seems unlikely, and if your proposal is "put into a production a system that you will have to spend $500/day every time it goes down and wait 2-4 business days for a resolution" then you're a braver person than I am.

Obviously not. You don't pay someone to just set it up. You pay to help do what you'd do if you had a dedicated devOps teams. You pay someone to set up the system with your team so they understand it , train your team to use it, write some documentation about it, script a rollback procedure, maybe help on developing playbooks, etc. Besides, there are people out there who offer on-call services for a small retainer.

This is optimistic to say the least. I've worked as an SRE for 5 years and apart from the others in the team the devs don't have nearly as much knowledge. There's no way I'd rely on them to fix an outage.

And even on a small retainer you'd better hope they retained the knowledge of how all that stuff works if you're only calling on them every now and again.

Post reply on HN