[dead]
Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
71–80 of 133 posts
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#72Can someone briefly explain how ZDD works in general? I guess both versions of the app must be running simultaneously, with new traffic being routed to the new version of the app. But what about DB migrations? Assuming the app uses a single database, and the new version of the app introduces changes to the DB schema, the new app version would modify the schema during startup via a migration script. However, the previ…
Others have described the how part if you do need truly zero downtime deployments, but I think it's worth pointing out that for most organisations, and most migrations, the amount of downtime due to a db migration is virtually indistinguishable from zero, particularly if you have a regional audience, and can aim for "quiet" hours to perform deployments.
Most are not affected by db migrations in the sense that migrations are run before the service starts the web server during boot. the database might block traffic for other already running connections though,in which case you have a problem with your database design.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#73I don't understand how to use this, maybe I am missing something. Following the example, it starts 4 replicas of a 'web' service. You can create a service by running a deploy to one of the replicas, let's say example-web-1. What does the other 3 replicas do? Now, let's say I update 'web'. Let's assume I want to do a zero-downtime deployment. That means I should be able to run a build command on the 'web' service, sta…
For the first part of your question about the other replicas, docker will load balance between all of the replicas either with a VIP or by returning multiple IPs in the DNS request[0]. I didn't check if this proxy balances across multiple records returned in a DNS request but, at least in the case of VIP-based load balancing, should work like you would expect. For the second part about updating the service, I'm a lit…
[deleted]
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#74This primarily exists to take care of a fundamental issue in Docker Swarm (Kamal's orchestrator of choice) where replacing containers of a service disrupts traffic. We had the same problem (when building JAMStack servers at Cloud 66) and used Caddy instead of writing our own proxy and also looked at Traefik which would have been just as suitable. I don't know why Kamal chose Swarm over k8s or k3s (simplicity perhaps?…
I feel like you’re conflating the orchestration with proxying. There’s no reason they couldn’t be using caddy or traefik or envoy for the proxy (just like k8s ends up using them as an ingress controller), while still using docker.
Docker is the container engine. Swarm is the orchestration, the same as Kubernetes. The concept of "Service" in k8s takes care of a lot of the proxying, while still using Docker (not anymore tho). In Swarm, services exist but only take care of container lifecycle and not traffic. While networking is left to the containers, Swarm services always get in the way, causing issues that will require a proxy.
In k8s for example, you can use Docker and won't need a proxy for ZDD (while you might want one for Ingress and other uses)
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#75This primarily exists to take care of a fundamental issue in Docker Swarm (Kamal's orchestrator of choice) where replacing containers of a service disrupts traffic. We had the same problem (when building JAMStack servers at Cloud 66) and used Caddy instead of writing our own proxy and also looked at Traefik which would have been just as suitable. I don't know why Kamal chose Swarm over k8s or k3s (simplicity perhaps?…
Kamal feels built around the premise that "Kubernetes is too complicated" (after Basecamp got burned by some hired help), and from that justification it goes out and recreates a sizable chunk of the things Kubernetes does. Your list of things a reverse proxy might do is a good example to me of how I expect this to go: what starts out as an ambition to be simple inevitably has to grow & grow more of complexity it soug…
I agree with you and at the risk of self-promotion, that's why we built Cloud 66 (which takes care of Day-1 (build and deploy) as well as Day-2 (scale and maintenance) part of infrastructure. As we all can see there is a lot to this than just wrapping code in a Dockerfile and pushing it out to a Swarm cluster.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#76It's an interesting choice to make this a whole app, when the zero-downtime deployments can be achieved with other servers trivially these days. For example any app+web proxy which supports Unix sockets can do zero-downtime by moving the file. It's atomic and you can send the warm-up requests with curl. Building a whole system with registration feels like an overkill.
That's just a small part of Kamal (https://kamal-deploy.org), their deployment tool they build and used to move from the cloud to their own hardware, saving millions (https://basecamp.com/cloud-exit).
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#77Does this implement the “traffic pausing” pattern? That’s where you have a proxy which effectively pauses traffic for a few seconds - incoming requests appear to take a couple of seconds longer than usual, but are still completed after that short delay. During those couple of seconds you can run a blocking infrastructure change - could be a small database migration, or could be something a little more complex as long…
tbh, sounds like "living dangerously" pattern to me.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#78Also exciting that Kamal 2 (currently RC https://github.com/basecamp/kamal/releases ) will support auto-SSL and make it easy to run multiple apps on one server with Kamal.
They're using the autocert package which is the bare minimum. It's brittle, doesn't allow for horizontal scaling of your proxy instances because you're subject to Let's Encrypt rate limits and simultaneous cert limits. (Disclaimer: I help maintain Caddy) Caddy/Certmagic solves this by writing the data to a shared storage so only a single cert will be issued and reused/coordinated across all instances through the storage. It also doesn't have issuer fallback, doesn't do rate limit avoidance, doesn't respect ARI, etc.
Holding requests until an upstream is available is also something Caddy does well, just configure the reverse_proxy with try_duration and try_interval, it will keep trying to choose a healthy upstream (determined via active health checks done in a separate goroutine) for that request until it times out.
Their proxy headers handling doesn't consider trusted IPs so if enabled, someone could spoof their IP by setting X-Forwarded-For. At least it's off by default, but they don't warn about this.
This looks pretty undercooked. I get that it's simple and that's the point, but I would never use this for anything in its current state. There's just so many better options out there.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#79Also exciting that Kamal 2 (currently RC https://github.com/basecamp/kamal/releases ) will support auto-SSL and make it easy to run multiple apps on one server with Kamal.
They're using the autocert package which is the bare minimum. It's brittle, doesn't allow for horizontal scaling of your proxy instances because you're subject to Let's Encrypt rate limits and simultaneous cert limits. (Disclaimer: I help maintain Caddy) Caddy/Certmagic solves this by writing the data to a shared storage so only a single cert will be issued and reused/coordinated across all instances through the stor…
I just want to say I use Caddy for this exact thing and it works beautifully! :D Thank you all for your work!
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#803 years from now they'll have invented their own AWS. NIH syndrome in full swing.
It's a matter of cost, not NIH syndrome. In Basecamp's case, saving $2.4M a year isn't something to ignore. https://basecamp.com/cloud-exit Of course, it's fair to say that rebuilding the components that the industry uses for hosting on bare metal is NIH syndrome.
A new proxy is a proxy filled with issues. It's nice that it's go, but in production I'd go with nginx or something else and replay traffic to kamal. There are enough weird behaviors out there (and bad actors) that I'd be worried about exploits etc.