291–300 of 589 posts
Earlier quoted context omitted.
I worked at google. k8s does not really look at all like what they used internally when I was there, aside from sharing some similar looking building blocks.
Yeah, but is the internal tool simpler? I'd be surprised.
The point about VMs being the wrong shape because they’re tied to CPU/memory resonates hard. The abstraction forces you to pay for time, not work. I ended up buying a cheap auctioned Hetzner server and using my self-hostable Firecracker orchestrator on top of it ( https://github.com/sahil-shubham/bhatti , https://bhatti.sh ) specifically because I wanted the thing he’s describing — buy some hardware, carve it into as…
If you want to run a website in the cloud, you start with an API, right? A CRUD API with commands like "make me a VPC with subnet 1.2.3.4/24", "make me a VM with 2GB RAM and 1 vCPU", "allow tcp port 80 and 443 to my VM", etc. Over time you create and change more things; things work, everybody's happy. At some point, one of the things changes, and now the website is broken. You could use Terraform or Ansible to try to fix this, by first creating all the configs to hopefully be in the right state, then re-running the IaC to re-apply the right set of parameters. But your website is already down and you don't really want to maintain a complex config and tool.
You can't avoid this problem because the cloud's design is bad. The CRUD method works at first to get things going. But eventually VMs stop, things get deleted, parameters of resources get changed. K8s was (partly) made to address this, with a declarative config and server which constantly "fixes" the resources back to the declared state. But K8s is hell because it uses a million abstractions to do a simple thing: ensure my stuff stays working. I should be able to point and click to set it up, and the cloud should remember it. Then if I try to change something like the security group, it should error saying "my dude, if you remove port 443 from the security group, your website will go down". Of course the cloud can't really know what will break what, unless the user defines their application's architecture. So the cloud should let the user define that architecture, have a server component that keeps ensuring everything's there and works, and stops people from footgunning themselves.
Everything that affects the user is a distributed system with mutable state. When that state changes, it can break something. So the system should continuously manage itself to fix issues that could break it. Part of that requires tracking dependencies, with guardrails to determine if a change might break something. Another part requires versioning the changes, so the user (or system) can easily roll back the whole system state to before it broke. This abstraction is complicated, but it's a solution to a complex problem: keeping the system working.
No cloud deals with this because it's too hard. But your cloud is extremely simple, so it might work. Ideally, every resource in your cloud (exe.dev) should work this way. From your team membership settings, to whether a proxy is public, the state of your VM, your DNS settings, the ssh keys allowed, email settings, http proxy integration / repo integration settings / their attachments, VM tags & disk sizes, etc. Over time your system will add more pieces and get more complex, to the point that implementing these system protections will be too complex and you won't even consider it. But your system is small right now, so you might be able to get it working. The end result should be less pain for the user because the system protects them from pain (fixing broken things, preventing breaking things), and more money for you because people like systems that don't break. But it's also possible nobody cares about this stuff until the system gets really big, so maybe your users won't care. It would be nice to have a cloud that fixes this tho.
Earlier quoted context omitted.
In some sense, Kubernetes is just a portable platform for running Linux services, even on a single node using something like K3s. I almost see it as being an extension of the Linux OS layer.
Then why can't we put a wrapper onto systemd and make that into a light weight k8s?
> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…
I don't work that closely with k8s, but have toyed with a cluster in my homelab, etc. Way back before it really got going, I observed some OpenStack folks make the jump to k8s.
Knowing what I knew about OpenStack, that gave me an inkling that what you describe would happen and we'd end up in this place where a reasonable thing exists but it has all of this crud layered on top. There are places where k8s makes sense and works well, but the people surrounding any project are the most important factor in the end result.
Today we have an industry around k8s. It keeps a lot of people busy and employed. These same folks will repeat k8s the next time, so the best thing people that who feel they have superior taste is to press forward with their own ideas as the behavior won't change.
Perhaps the VM idea is old. The unit is a worker encapsulated in some deployable container.
In the world of Cloudflare workers - especially durable objects that are guaranteed to have one of them running in the world with a tightly bound database.
The way I think of apps has changed.
My take is devs want a way to say “run this code, persist this info, microsecond latency, never go down, scale within this $ budget”
It’s crazy how good a deal $5/mo cloudflare standard plan is.
Obviously many startups raise millions and they gotta spend millions.
However the new age of scale to zero, wake up in millisecond, process the request and go back to sleep is a new paradigm.
Vs old school of over provision for max capacity you will ever need.
Google has a similar, scale to zero container story but their cold startup time is in seconds. Too slow.
Earlier quoted context omitted.
If you spin up Kubernetes for "a couple of containers to run your web app", I think you're doing something wrong in the first place, also coupled with your comment about adding SDN to Kubernetes. People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes.
Depends. For personal projects, yeah definitely. But at work? Typically the “Platform” team can only afford to support 1 (maybe 2) ways of deployment, and k8s is quite versatile, so even if you need 1 small service, you’ll go with the self-service-k8s approach your Platform team offers. Because the alternative is for you (or your team) to own the whole infrastructure stack for your new deloyment model (ecs? lambda? W…
I like to think if we had a K8s environment a lot of this would be built out within it. Having that functionality abstracted away from the developer would be a huge win in my opinion.
> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…
We've reduced our costs on Hetzner to about 10% on what we've paid on Heroku, for 10x performance. Kamal really kicks ass, and you can have a pretty complicated infrastructure up in no time. We're using terraform, ansible + kamal for deploys, no issues whatsoever.