Every month, there seems to be a new post stating ~"don't use Kubernetes yet", but what should we use instead if we cannot have downtime?
Ask HN: How to do rolling deployments without Kubernetes?
1–10 of 19 posts
Re: Ask HN: How to do rolling deployments without Kubernetes?
#2I don’t know much about kubernetes (I suspect I may soon since our current process annoys me) and I’m not a DevOps engineer (but we don’t have any, and any consultants we’ve tried is basically just a little more confident using the official documentation than we are).
Anyway, how we’ve done it is by setting up a pipeline, that builds and deploys a docker container of a repository to a deployment a lot on a cloud application/function/whatever you call them if it passes the build phase. Then depending on whether or not the deployment passes the criteria set up for it, it either automatically swaps deployment slots with the production slot or waits for manual confirmation to do so. On anything that is allowed a reload, we don’t swap deployment slots but instead redeploy, typically with minimal downtime.
The reason it’s annoying is because it takes a lot of time to set up for each pipeline, and when you need to be able to swap deployment slots with now downtime you also need to handle things like build processes taking global variables or clients needing to be told to reload parts or them.
I’m not sure kubernetes is the answer for us, but we’re certainly going to look for a way to make the whole process smarter as it sometimes takes up more time to setup the pipeline and deployment environment than building the service that needs to be deployed.
Re: Ask HN: How to do rolling deployments without Kubernetes?
#3This article provides one possible approach (and includes links to an example implementation): https://blog.jakubholy.net/2013/09/05/blue-green-deployment-...
Re: Ask HN: How to do rolling deployments without Kubernetes?
#4I use Cloud Run which has all of this built into it by default.
https://cloud.google.com/blog/products/serverless/cloud-run-...
Re: Ask HN: How to do rolling deployments without Kubernetes?
#5Instead of talking about specific technologies/software/services, I'll give a quick rundown how you can achieve this in theory, and hopefully it can apply to whatever you're currently using.
The requirements:
- Your application should be able to run on arbitrary ports, preferably controlled with env vars or similar
- You need to have something in front of your application, like nginx or apache
- Whatever webserver you have should be able to "hang"/pause/suspend requests while you switch application version
- Each version you create needs to be concerned about what the previous version did. Breaking changes needs to happen across multiple versions, where you can soft-deprecate something, and a release after that, actually "break" it
- You need to have some sort of "healthchecking" that can tell you if the new version is OK or not
The implementation:
- You have Version 1 running of your backend on port X
- You want to deploy the new version, so you deploy it to your server but the web server in front still serves requests from Version 1
- Run healthchecks against Version 2
- Once they pass, tell web server to "pause" in-flight requests
- Switch web server configuration to use Version 2 application instead (this can be combined with the previous step, `nginx reload` would combine these for example)
- Stop Version 1 from running
- Repeat for each new version
And now you've achieved deploying a new version of your application without any failing requests.
(Sidenote: You can replace hanging/pausing requests with graceful shutdown of your webserver (meaning it waits for no pending requests) if you have a low amount of traffic)
Re: Ask HN: How to do rolling deployments without Kubernetes?
#6Use container registries and automatic pull deployment. AWS fargate, Azure App Services both allow this method of working.
I.e. github action to do your build, test. Dont do the docker push to the repository unless it passes the tests. Always do your docker push from the correct branch. If you need approval for release, build that into the pipeline.
Azure pipelines pretty much the same deal.
You don't need kubernetes for a few services, it only gets useful when you have a lot.
Re: Ask HN: How to do rolling deployments without Kubernetes?
#7Assuming we're talking over-HTTP traffic, you can use a load balancer in-front of your Kubernetes cluster.
We did this at work and works fine (blogpost will be out soon). It allows us to decommission a cluster by removing the cluster from the LB. There's nothing new about this technique.
We're using DOKS + CloudFlare Traffic but you can use any LB service (no affiliation, as all products there are pro's and con's).
Once the setup is ready, operations are easy:
a) Remove cluster from the LB
b) Perform cluster operations (ingress upgrade, k8s upgrade, possibly disruptive daemonset operations)
c) Add cluster to LB
Another pro is that when a region has cloud-provider level issues (happened with FRA1 few days ago) we can remove the cluster from the LB and stop worrying about it until the the issue is fixed. LBs have health-checks and such, to automate addition/remove of clusters.
Re: Ask HN: How to do rolling deployments without Kubernetes?
#8Ecs does this automatically https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
Re: Ask HN: How to do rolling deployments without Kubernetes?
#9Instead of talking about specific technologies/software/services, I'll give a quick rundown how you can achieve this in theory, and hopefully it can apply to whatever you're currently using. The requirements: - Your application should be able to run on arbitrary ports, preferably controlled with env vars or similar - You need to have something in front of your application, like nginx or apache - Whatever webserver yo…
> Instead of talking about specific technologies/software/services, I'll give a quick rundown how you can achieve this in theory, and hopefully it can apply to whatever you're currently using.
I wish more people approached their software engineering teaching in this way.
Thank you for the great comment.
Re: Ask HN: How to do rolling deployments without Kubernetes?
#10Instead of talking about specific technologies/software/services, I'll give a quick rundown how you can achieve this in theory, and hopefully it can apply to whatever you're currently using. The requirements: - Your application should be able to run on arbitrary ports, preferably controlled with env vars or similar - You need to have something in front of your application, like nginx or apache - Whatever webserver yo…
While this is right, I'm not sure why you mentioned pausing requests. If you can run v1 and v2 at the same time, you could switch where the new requests are going without affecting the old ones. There's many ways to do that, but what I do is point nginx upstream at a path which is a symlink to a Unix socket of a specific version. So it's more like: start the new version, check, update symlink, wait until old connections drop, kill old version.