Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

github.com

Post reply on HN
Print view View original

101–110 of 133 posts

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#101

Post by mwcampbell »

francislavoie wrote: post #78
shafyy wrote: post #8

Also exciting that Kamal 2 (currently RC https://github.com/basecamp/kamal/releases ) will support auto-SSL and make it easy to run multiple apps on one server with Kamal.

They're using the autocert package which is the bare minimum. It's brittle, doesn't allow for horizontal scaling of your proxy instances because you're subject to Let's Encrypt rate limits and simultaneous cert limits. (Disclaimer: I help maintain Caddy) Caddy/Certmagic solves this by writing the data to a shared storage so only a single cert will be issued and reused/coordinated across all instances through the stor…

I doubt that 37signals themselves are using Kamal's new automatic SSL support, at least in their main apps, since they're running their apps behind Cloudflare. I'd love to see a deployment tool that's like Kamal, in that it's simpler than Kubernetes, but fronted by something mature like Caddy, that can be exposed directly to the Internet rather than running behind a CDN like Cloudflare.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#102

Post by chipdart »

ksajadi wrote: post #55

This primarily exists to take care of a fundamental issue in Docker Swarm (Kamal's orchestrator of choice) where replacing containers of a service disrupts traffic. We had the same problem (when building JAMStack servers at Cloud 66) and used Caddy instead of writing our own proxy and also looked at Traefik which would have been just as suitable. I don't know why Kamal chose Swarm over k8s or k3s (simplicity perhaps?…

> This primarily exists to take care of a fundamental issue in Docker Swarm (Kamal's orchestrator of choice) where replacing containers of a service disrupts traffic.

If that's the reason then I feel someone failed to do even the most cursory research on the topic.

I mean, a staple of any container orchestration service is the use of an ingress controller to handle dialups and blue/green deployments.

More to the point, ingress controllers is the name given to the role assigned to any of the myriad reverse proxies readily available for years, if not decades, to handle this very same usecase.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#103

Post by chipdart »

jauntywundrkind wrote: post #62
ksajadi wrote: post #55

This primarily exists to take care of a fundamental issue in Docker Swarm (Kamal's orchestrator of choice) where replacing containers of a service disrupts traffic. We had the same problem (when building JAMStack servers at Cloud 66) and used Caddy instead of writing our own proxy and also looked at Traefik which would have been just as suitable. I don't know why Kamal chose Swarm over k8s or k3s (simplicity perhaps?…

Kamal feels built around the premise that "Kubernetes is too complicated" (after Basecamp got burned by some hired help), and from that justification it goes out and recreates a sizable chunk of the things Kubernetes does. Your list of things a reverse proxy might do is a good example to me of how I expect this to go: what starts out as an ambition to be simple inevitably has to grow & grow more of complexity it soug…

> But a huge part of me sees Kamal & thinks, man, this is a lot of work being done only to have to keep walking backwards into the complexity they were trying to avoid.

Fully agree. I get resume-driven development vibes out of this project.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#104

Post by chipdart »

stackskipton wrote: post #82
yla92 wrote: post #20

Earlier quoted context omitted.

They were actually using Traefik until this "v2.0.0" (pre-release right now) version. There are some context about why they switched and decided to roll their own, from the PR. https://github.com/basecamp/kamal/pull/940

As SRE, that PR scares me. There is no long explanation of why we are throwing out third party, extremely battle tested HTTP Proxy software for our own homegrown except "Traefik didn't do what we wanted 100%". Man, I've been there where you wish third party software had some feature but writing your own is WORST thing you can do for a company 9/10 times. My current company is dealing with massive tech debt because of…

> Man, I've been there where you wish third party software had some feature (...)

It looks to me that they were already using Traefik, and Traefik does indeed support the feature they wished to have.

Apparently they also got rid of support for healthchecks, which all by itself would be expected to require an extensive justification. But no.

I would love to hear why the people behind this decision felt that Traefik's support for dynamic route configuration wasn't suited to configure routes dynamically.

I would also love to hear the rationale behind the decision to invest their time rolling their own reverse proxy instead of just doing docker run on one of the many production-grade reverse proxies readily available.

I don't know. This has vibes of spending months in a development project to avoid spending hours reading a manual.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#105

Post by chipdart »

francislavoie wrote: post #94
moondev wrote: post #28

Does this handle a host reboot?

In theory it should, because they do health checking to track status of the upstreams. The upstream server being down would be a failed TCP connection which would fail the health check. Obviously, rebooting the machine the proxy is running on is trickier though. I don't feel confident they've done enough to properly support having multiple proxy instances running side by side (no shared storage mechanism for TLS cert…

> In theory it should, because they do health checking to track status of the upstreams.

I think the PR that pushes this kamal-proxy project explicitly removes supports for healthchecks? So it's unclear.

In theory, a reverse proxy like Traefik supports this feature.

In practice it does too.

So I don't know. It seems there's some rationale that's definitely missing from the whole story. I doubt people haphazardly decide to roll out a custom reverse proxy developed in-house. The reasons laid out in the doc definitely don't seem to be it.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#106

Post by chipdart »

martinald wrote: post #9
oDot wrote: post #2

DHH mentioned they built it to move from the cloud to bare metal. He glorifies the simplicity but I can't help thinking they are a special use case of predictable, non-huge load. Uber, for example, moved to the cloud. I feel like in the span between them there are far more companies for which Kamal is not enough. I hope I'm wrong, though. It'll be nice for many companies to be have the choice of exiting the cloud.

I don't think that's the real point. The real point is that 'big 3' cloud providers are so overpriced that you could run hugely over provisioned infra 24/7 for your load (to cope with any spikes) and still save a fortune. The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is. Slow single thread performance because of using the most parallel CPUs p…

> I don't think that's the real point. The real point is that 'big 3' cloud providers are so overpriced that you could run hugely over provisioned infra 24/7 for your load (to cope with any spikes) and still save a fortune.

You don't need to roll out your own reverse proxy project to run services in-house.

Any container orchestration service was designed for that scenario. It's why they exist.

Under the hood, applications include a reverse proxy to handle deployment scenarios, like blue/green, onebox, canary, etc.

You definitely do not need to roll your own project to do that.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#107

Post by chipdart »

sgarland wrote: post #60
martinald wrote: post #9

Earlier quoted context omitted.

I don't think that's the real point. The real point is that 'big 3' cloud providers are so overpriced that you could run hugely over provisioned infra 24/7 for your load (to cope with any spikes) and still save a fortune. The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is. Slow single thread performance because of using the most parallel CPUs p…

> The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is. This. Mostly disk latency, for me. People who have only ever known DBaaS have no idea how absurdly fast they can be when you don’t have compute and disk split by network hops, and your disks are NVMe. Of course, it doesn’t matter, because the 10x latency hit is overshadowed by the miasma of…

> Of course, it doesn’t matter, because the 10x latency hit is overshadowed by the miasma of everything else in a modern stack.

This. Those complaining about performance seem to come from people who are not be aware of latency numbers.

Sure, the latency from reading data from a local drive can be lower than 1ms, whereas in block storage services like AWS EBS it can take more than 10ms. An order of magnitude slower. Gosh, that's a lot.

But whatever your disk access needs, your response will be sent over the wire to clients. That takes between 100-250ms.

Will your users even notice a difference if your response times are 110ms instead of 100ms? Come on.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#108

Post by rendaw »

irundebian wrote: post #7
000ooo000 wrote: post #3

Strange choice of language for the actions: >To route traffic through the proxy to a web application, you *deploy* instances of the application to the proxy. *Deploying* an instance makes it available to the proxy, and replaces the instance it was using before (if any). >e.g. `kamal-proxy deploy service1 --target web-1:3000` 'Deploy' is a fairly overloaded term already. Fun conversations ahead. Is the app deployed? Y…

Deployed = running and registered to the proxy.

GP isn't claiming deployed is incorrect.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#109

Post by m_sahaf »

francislavoie wrote: post #92
simonw wrote: post #70

Does this implement the “traffic pausing” pattern? That’s where you have a proxy which effectively pauses traffic for a few seconds - incoming requests appear to take a couple of seconds longer than usual, but are still completed after that short delay. During those couple of seconds you can run a blocking infrastructure change - could be a small database migration, or could be something a little more complex as long…

Caddy does this! (As you know, I think. I feel I remember we discussed this some time ago)

Yep, it was Simon: https://til.simonwillison.net/caddy/pause-retry-traffic

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#110

Post by dewey »

chipdart wrote: post #100
dewey wrote: post #76

Earlier quoted context omitted.

That's just a small part of Kamal ( https://kamal-deploy.org ), their deployment tool they build and used to move from the cloud to their own hardware, saving millions ( https://basecamp.com/cloud-exit ).

> That's just a small part of Kamal (...) That does not address OP's point. Any plain old reverse proxy handles this usecase, particularly those already used extensively as ingress controllers for container orchestration systems. Some even support specialized routing strategies. Other than NIH, what exactly does kamal-proxy offers that any other reverse proxy doesn't already provide?

Ah, I was more referring to why this tool was released and wanted to say it’s a component of their Kamal deployment tool and not so important on its own.

There’s some more explanation on why they decided to switch away from Traefik:

https://github.com/basecamp/kamal/pull/940

Post reply on HN