Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

github.com

Post reply on HN
Print view View original

61–70 of 133 posts

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#61

Post by igortg »

martinald wrote: post #9
oDot wrote: post #2

DHH mentioned they built it to move from the cloud to bare metal. He glorifies the simplicity but I can't help thinking they are a special use case of predictable, non-huge load. Uber, for example, moved to the cloud. I feel like in the span between them there are far more companies for which Kamal is not enough. I hope I'm wrong, though. It'll be nice for many companies to be have the choice of exiting the cloud.

I don't think that's the real point. The real point is that 'big 3' cloud providers are so overpriced that you could run hugely over provisioned infra 24/7 for your load (to cope with any spikes) and still save a fortune. The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is. Slow single thread performance because of using the most parallel CPUs p…

I'm using a managed Postgres instance in a well known provider and holy shit, I couldn't believe how slow it is. For small datasets I couldn't notice, but when one of the tables reached 100K rows, queries started to take 5-10 seconds (the same query takes 0.5-0.6 in my standard i5 Dell laptop).

I wasn't expecting blasting speed on the lowest tear, but 10x slower is bonkers.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#62

Post by jauntywundrkind »

ksajadi wrote: post #55

This primarily exists to take care of a fundamental issue in Docker Swarm (Kamal's orchestrator of choice) where replacing containers of a service disrupts traffic. We had the same problem (when building JAMStack servers at Cloud 66) and used Caddy instead of writing our own proxy and also looked at Traefik which would have been just as suitable. I don't know why Kamal chose Swarm over k8s or k3s (simplicity perhaps?…

Kamal feels built around the premise that "Kubernetes is too complicated" (after Basecamp got burned by some hired help), and from that justification it goes out and recreates a sizable chunk of the things Kubernetes does.

Your list of things a reverse proxy might do is a good example to me of how I expect this to go: what starts out as an ambition to be simple inevitably has to grow & grow more of complexity it sought to avoid.

Part of me strongly thinks we need competition & need other things trying to create broad ideally extensible ways or running systems. But a huge part of me sees Kamal & thinks, man, this is a lot of work being done only to have to keep walking backwards into the complexity they were trying to avoid. Usually second system syndrome is the first system being simple the second being overly complicated, and on the tin the case is inverse, but man, the competency of Kube & it's flexibility/adaptability as being a framework for Desired State Management really shows through for me.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#63

Post by hipadev23 »

ksajadi wrote: post #55

This primarily exists to take care of a fundamental issue in Docker Swarm (Kamal's orchestrator of choice) where replacing containers of a service disrupts traffic. We had the same problem (when building JAMStack servers at Cloud 66) and used Caddy instead of writing our own proxy and also looked at Traefik which would have been just as suitable. I don't know why Kamal chose Swarm over k8s or k3s (simplicity perhaps?…

I feel like you’re conflating the orchestration with proxying. There’s no reason they couldn’t be using caddy or traefik or envoy for the proxy (just like k8s ends up using them as an ingress controller), while still using docker.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#65

Post by lukevp »

sgarland wrote: post #59
diggan wrote: post #38

Earlier quoted context omitted.

> the amount of downtime due to a db migration is virtually indistinguishable from zero Besides, once you've run a service for a while that has acquired enough data for migrations to take a while, you realize that there are in fact two different types of migrations. "Schema migrations" which are generally fast and "Data migrations" that depending on the amount of data can take seconds or days. Or you can do the "data…

Schema migrations can be quite lengthy, mostly if you made a mistake earlier. Some things that come to mind are changing a column’s type, or extending VARCHAR length (with caveats; under certain circumstances it’s instant).

Not OP, but I would consider this a data migration as well. Anything that requires an operation on every row in a table would qualify. Really changing the column type is just a built in form of a data migration.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#66

Post by shipp02 »

diggan wrote: post #17
blue_pants wrote: post #12

Can someone briefly explain how ZDD works in general? I guess both versions of the app must be running simultaneously, with new traffic being routed to the new version of the app. But what about DB migrations? Assuming the app uses a single database, and the new version of the app introduces changes to the DB schema, the new app version would modify the schema during startup via a migration script. However, the previ…

First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with…

So if you add any constraints/data, you can't rely on them being there until version n+2 or you need to have 2 paths 1 for the old date, 1 for new?

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#67

Post by whartung »

wejick wrote: post #31
diggan wrote: post #17

Earlier quoted context omitted.

First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with…

This is very good explanation, no judgment and simply educational. Appreciated Though I'm still surprised that some people run DB alteration on application start up. Never saw one in real life.

We do this. It has worked very well for us.

There's a couple of fundamental rules to follow. First, don't put something that will have insane impact into the application deploy changes. 99% of the DB changes are very cheap, and very minor. If the deploy is going to be very expensive, then just don't do it, we'll do it out of band. This has not been a problem in practice with our 20ish person team.

Second, it was kind of like double entry accounting. Once you committed the change, you can not go back and "fix it". If you did something really wrong (i.e. see above), then sure, but if not, you commit a correcting entry instead. Because you don't know who has recently downloaded your commit, and run it against their database.

The changes are a list of incremental steps that the system applies in order, if they had not been applied before. So, they are treated as, essentially, append only.

And it has worked really well for us, keeping the diverse developers who deploy again local databases in sync with little drama.

I've incorporated the same concept in my GUI programs that stand up their own DB. It's a very simple system.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#68

Post by e_y_ »

wejick wrote: post #31
diggan wrote: post #17

Earlier quoted context omitted.

First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with…

This is very good explanation, no judgment and simply educational. Appreciated Though I'm still surprised that some people run DB alteration on application start up. Never saw one in real life.

At my company, DB migrations on startup was a flag that was enabled for local development and disabled for production deploys. Some teams had it enabled for staging/pre-production deploys, and a few teams had it turned on for production deploys (although those teams only had infrequent, minor changes like adding a new column).

Personally I found the idea of having multiple instances running the same schema update job at the same time (even if locks would keep it from running in practice) to be concerning so I always had it disabled for deploys.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#69

Post by simonw »

shipp02 wrote: post #66
diggan wrote: post #17

Earlier quoted context omitted.

First step is to decouple migrations from deploys, you want manual control over when the migrations run, contrary to many frameworks default of running migrations when you deploy the code. Secondly, each code version has to work with the current schema and the schema after a future migration, making all code effectively backwards compatible. Your deploys end up being something like: - Deploy new code that works with…

So if you add any constraints/data, you can't rely on them being there until version n+2 or you need to have 2 paths 1 for the old date, 1 for new?

Effectively yes. Zero downtime deployments with database migrations are fiddly.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#70

Post by simonw »

Does this implement the “traffic pausing” pattern?

That’s where you have a proxy which effectively pauses traffic for a few seconds - incoming requests appear to take a couple of seconds longer than usual, but are still completed after that short delay.

During those couple of seconds you can run a blocking infrastructure change - could be a small database migration, or could be something a little more complex as long as you can get it finished in less than about 5 seconds.

Post reply on HN