Earlier quoted context omitted.
Then wouldn’t “register” be a better term?
It’s registered but is it deployed?
Who'd a thunk it?
Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
81–90 of 133 posts
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#82Did they mention anywhere why they decided to write their own proxy instead of using Traefik or something else battle tested?
They were actually using Traefik until this "v2.0.0" (pre-release right now) version. There are some context about why they switched and decided to roll their own, from the PR. https://github.com/basecamp/kamal/pull/940
As SRE, that PR scares me. There is no long explanation of why we are throwing out third party, extremely battle tested HTTP Proxy software for our own homegrown except "Traefik didn't do what we wanted 100%".
Man, I've been there where you wish third party software had some feature but writing your own is WORST thing you can do for a company 9/10 times. My current company is dealing with massive tech debt because of all this homegrown software.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#83Does this implement the “traffic pausing” pattern? That’s where you have a proxy which effectively pauses traffic for a few seconds - incoming requests appear to take a couple of seconds longer than usual, but are still completed after that short delay. During those couple of seconds you can run a blocking infrastructure change - could be a small database migration, or could be something a little more complex as long…
Have you seen that done in production? It sounds really dangerous, I've worked for an app server company for years and this is the first I've heard of this pattern. I'd wave it away if I didn't notice in your bio that you co-created Django so you've probably seen your fair share of deployments.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#84Strange choice of language for the actions: >To route traffic through the proxy to a web application, you *deploy* instances of the application to the proxy. *Deploying* an instance makes it available to the proxy, and replaces the instance it was using before (if any). >e.g. `kamal-proxy deploy service1 --target web-1:3000` 'Deploy' is a fairly overloaded term already. Fun conversations ahead. Is the app deployed? Y…
“Yo dawg, i heard you like deployments, so we deployed a deployment in your deployment so your deployment can deploy” -Xzibit
"Pimp my deployment!"
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#85Does this implement the “traffic pausing” pattern? That’s where you have a proxy which effectively pauses traffic for a few seconds - incoming requests appear to take a couple of seconds longer than usual, but are still completed after that short delay. During those couple of seconds you can run a blocking infrastructure change - could be a small database migration, or could be something a little more complex as long…
Have you seen that done in production? It sounds really dangerous, I've worked for an app server company for years and this is the first I've heard of this pattern. I'd wave it away if I didn't notice in your bio that you co-created Django so you've probably seen your fair share of deployments.
Just asking, isn't this what every serverless platform uses while it spins up an instance? Like it's why cold starts are a topic at all, or else the first few requests would just fail until the instance spun up to handle the request.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#86Also exciting that Kamal 2 (currently RC https://github.com/basecamp/kamal/releases ) will support auto-SSL and make it easy to run multiple apps on one server with Kamal.
They're using the autocert package which is the bare minimum. It's brittle, doesn't allow for horizontal scaling of your proxy instances because you're subject to Let's Encrypt rate limits and simultaneous cert limits. (Disclaimer: I help maintain Caddy) Caddy/Certmagic solves this by writing the data to a shared storage so only a single cert will be issued and reused/coordinated across all instances through the stor…
Interesting, thanks for the detailed explanation! I'm not very experienced with devops, so this is very helpful!
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#87Does this implement the “traffic pausing” pattern? That’s where you have a proxy which effectively pauses traffic for a few seconds - incoming requests appear to take a couple of seconds longer than usual, but are still completed after that short delay. During those couple of seconds you can run a blocking infrastructure change - could be a small database migration, or could be something a little more complex as long…
Have you seen that done in production? It sounds really dangerous, I've worked for an app server company for years and this is the first I've heard of this pattern. I'd wave it away if I didn't notice in your bio that you co-created Django so you've probably seen your fair share of deployments.
I first heard about it from Braintree. https://simonwillison.net/2011/Jun/30/braintree/
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#88Can someone briefly explain how ZDD works in general? I guess both versions of the app must be running simultaneously, with new traffic being routed to the new version of the app. But what about DB migrations? Assuming the app uses a single database, and the new version of the app introduces changes to the DB schema, the new app version would modify the schema during startup via a migration script. However, the previ…
First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with…
There's a little bit more to it. Firstly you can deploy the migration first as long as it's forwards compatible (ie. old code can read from it). That migration needs to be zero downtime; it can't, for example, rewrite whole tables or otherwise lock them, or requests will time out. Doing a whole new schema is one way to do it, but not always necessary. In any case you probably then need a backfill job to fill up the new schema with data before possibly removing the old one.
There's a good post about it here: https://rtpg.co/2021/06/07/changes-checklist.html
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#89Earlier quoted context omitted.
Others have described the how part if you do need truly zero downtime deployments, but I think it's worth pointing out that for most organisations, and most migrations, the amount of downtime due to a db migration is virtually indistinguishable from zero, particularly if you have a regional audience, and can aim for "quiet" hours to perform deployments.
> the amount of downtime due to a db migration is virtually indistinguishable from zero Besides, once you've run a service for a while that has acquired enough data for migrations to take a while, you realize that there are in fact two different types of migrations. "Schema migrations" which are generally fast and "Data migrations" that depending on the amount of data can take seconds or days. Or you can do the "data…
Lengthy migrations doesn't matter. What matters is whether they hold long locks or not. Data migrations might take a while but they won't lock anything. Schema migrations, on the other hand, can easily do so, like if you add a new column with a default value. The whole table must be rewritten and it's locked for the entire time.
Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments
#90Does this implement the “traffic pausing” pattern? That’s where you have a proxy which effectively pauses traffic for a few seconds - incoming requests appear to take a couple of seconds longer than usual, but are still completed after that short delay. During those couple of seconds you can run a blocking infrastructure change - could be a small database migration, or could be something a little more complex as long…
Have you seen that done in production? It sounds really dangerous, I've worked for an app server company for years and this is the first I've heard of this pattern. I'd wave it away if I didn't notice in your bio that you co-created Django so you've probably seen your fair share of deployments.
[deleted]