It's an interesting choice to make this a whole app, when the zero-downtime deployments can be achieved with other servers trivially these days. For example any app+web proxy which supports Unix sockets can do zero-downtime by moving the file. It's atomic and you can send the warm-up requests with curl. Building a whole system with registration feels like an overkill.
Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
51–60 of 133 posts
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#523 years from now they'll have invented their own AWS. NIH syndrome in full swing.
It's a matter of cost, not NIH syndrome. In Basecamp's case, saving $2.4M a year isn't something to ignore.
https://basecamp.com/cloud-exit
Of course, it's fair to say that rebuilding the components that the industry uses for hosting on bare metal is NIH syndrome.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#53Can someone briefly explain how ZDD works in general? I guess both versions of the app must be running simultaneously, with new traffic being routed to the new version of the app. But what about DB migrations? Assuming the app uses a single database, and the new version of the app introduces changes to the DB schema, the new app version would modify the schema during startup via a migration script. However, the previ…
First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with…
I thought it was correct to run the DB migrations for the new code first, then deploy the new code. While making sure that the DB schema is backwards compatible with both versions of the code that will be running during the deployment.
So maybe there's something I'm missing about running DB migrations after the new code has been deployed - could you explain?
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#54I don't understand how to use this, maybe I am missing something. Following the example, it starts 4 replicas of a 'web' service. You can create a service by running a deploy to one of the replicas, let's say example-web-1. What does the other 3 replicas do? Now, let's say I update 'web'. Let's assume I want to do a zero-downtime deployment. That means I should be able to run a build command on the 'web' service, sta…
For the first part of your question about the other replicas, docker will load balance between all of the replicas either with a VIP or by returning multiple IPs in the DNS request[0]. I didn't check if this proxy balances across multiple records returned in a DNS request but, at least in the case of VIP-based load balancing, should work like you would expect.
For the second part about updating the service, I'm a little less clear. I guess the expectation would be to bring up a differently-named service within the same network, and then `kamal-proxy deploy` it? So maybe the expectation is for service names to include a version number? Keeping the old version hot makes sense if you want to quickly be able to route back to it.
[0]: https://docs.docker.com/reference/compose-file/deploy/#endpo...
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#55This primarily exists to take care of a fundamental issue in Docker Swarm (Kamal's orchestrator of choice) where replacing containers of a service disrupts traffic. We had the same problem (when building JAMStack servers at Cloud 66) and used Caddy instead of writing our own proxy and also looked at Traefik which would have been just as suitable.
I don't know why Kamal chose Swarm over k8s or k3s (simplicity perhaps?) but then, complexity needs a home, you can push it around but cannot hide it, hence a home grown proxy.
I have not tried Kamal proxy to know, but I am highly skeptical of something like this, because I am pretty sure I will be chasing it for support for anything from WebSockets to SSE, to HTTP/3 to various types of compression and encryption.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#56Earlier quoted context omitted.
Why would I not just do k8s rollout restart deployment? Or just switch my DNS or router between two backends?
You still need some warm-up routine to run for the newly online server before the hand-off occurs. I'm not a k8s expert, but the above described events can be easily handled by a bash or fab script.
This is a health/readiness probe in k8s. It's already solved quite solidly.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#57I don't understand how to use this, maybe I am missing something. Following the example, it starts 4 replicas of a 'web' service. You can create a service by running a deploy to one of the replicas, let's say example-web-1. What does the other 3 replicas do? Now, let's say I update 'web'. Let's assume I want to do a zero-downtime deployment. That means I should be able to run a build command on the 'web' service, sta…
Why would I not just do k8s rollout restart deployment? Or just switch my DNS or router between two backends?
I think the parent project, Kamal, positions itself as a simpler alternative to K8s when deploying web apps. They have a question on this on their website: https://kamal-deploy.org
"Why not just run Capistrano, Kubernetes or Docker Swarm?
...
Docker Swarm is much simpler than Kubernetes, but it’s still built on the same declarative model that uses state reconciliation. Kamal is intentionally designed around imperative commands, like Capistrano.
Ultimately, there are a myriad of ways to deploy web apps, but this is the toolkit we’ve used at 37signals to bring HEY and all our other formerly cloud-hosted applications home to our own hardware."
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#58Earlier quoted context omitted.
First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with…
I thought it was correct to run the DB migrations for the new code first, then deploy the new code. While making sure that the DB schema is backwards compatible with both versions of the code that will be running during the deployment. So maybe there's something I'm missing about running DB migrations after the new code has been deployed - could you explain?
I'm not the person you've asked, but I've worked in devops before.
It kinda doesn't matter which you do first. And if you squint a little, it's effectively the same thing, because the migration will likely only become available via a deployment too
So yeah, the only things that's important is that the DB migration can't cause an incompatibility with any currently deployed version of the code - and if it would, you'll have to split the change so it doesn't. It'll force another deploy for the change you want to do, but it's what you're forced to do if maintenance windows aren't an option. Which is kinda a given for most b2c products
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#59Earlier quoted context omitted.
Others have described the how part if you do need truly zero downtime deployments, but I think it's worth pointing out that for most organisations, and most migrations, the amount of downtime due to a db migration is virtually indistinguishable from zero, particularly if you have a regional audience, and can aim for "quiet" hours to perform deployments.
> the amount of downtime due to a db migration is virtually indistinguishable from zero Besides, once you've run a service for a while that has acquired enough data for migrations to take a while, you realize that there are in fact two different types of migrations. "Schema migrations" which are generally fast and "Data migrations" that depending on the amount of data can take seconds or days. Or you can do the "data…
Schema migrations can be quite lengthy, mostly if you made a mistake earlier. Some things that come to mind are changing a column’s type, or extending VARCHAR length (with caveats; under certain circumstances it’s instant).
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#60DHH mentioned they built it to move from the cloud to bare metal. He glorifies the simplicity but I can't help thinking they are a special use case of predictable, non-huge load. Uber, for example, moved to the cloud. I feel like in the span between them there are far more companies for which Kamal is not enough. I hope I'm wrong, though. It'll be nice for many companies to be have the choice of exiting the cloud.
I don't think that's the real point. The real point is that 'big 3' cloud providers are so overpriced that you could run hugely over provisioned infra 24/7 for your load (to cope with any spikes) and still save a fortune. The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is. Slow single thread performance because of using the most parallel CPUs p…
> The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is.
This. Mostly disk latency, for me. People who have only ever known DBaaS have no idea how absurdly fast they can be when you don’t have compute and disk split by network hops, and your disks are NVMe.
Of course, it doesn’t matter, because the 10x latency hit is overshadowed by the miasma of everything else in a modern stack. My favorite is introducing a caching layer because you can’t write performant SQL, and your DB would struggle to deliver it anyway.