Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

github.com

Post reply on HN
Print view View original

111–120 of 133 posts

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#111

Post by francislavoie »

chipdart wrote: post #105
francislavoie wrote: post #94

Earlier quoted context omitted.

In theory it should, because they do health checking to track status of the upstreams. The upstream server being down would be a failed TCP connection which would fail the health check. Obviously, rebooting the machine the proxy is running on is trickier though. I don't feel confident they've done enough to properly support having multiple proxy instances running side by side (no shared storage mechanism for TLS cert…

> In theory it should, because they do health checking to track status of the upstreams. I think the PR that pushes this kamal-proxy project explicitly removes supports for healthchecks? So it's unclear. In theory, a reverse proxy like Traefik supports this feature. In practice it does too. So I don't know. It seems there's some rationale that's definitely missing from the whole story. I doubt people haphazardly deci…

I'm looking here https://github.com/basecamp/kamal-proxy/tree/main/internal/s... which is the code for their v2 proxy. Notice there's a health_check.go which makes requests in a goroutine and sets the health to true/false based on the HTTP status, via a consumer interface.

As I said elsewhere in this topic, this is all too basic IMO, (disclaimer: I'm a Caddy maintainer) Caddy does this all in a more robust way.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#113

Post by tinco »

written-beyond wrote: post #85
tinco wrote: post #83

Earlier quoted context omitted.

Have you seen that done in production? It sounds really dangerous, I've worked for an app server company for years and this is the first I've heard of this pattern. I'd wave it away if I didn't notice in your bio that you co-created Django so you've probably seen your fair share of deployments.

Just asking, isn't this what every serverless platform uses while it spins up an instance? Like it's why cold starts are a topic at all, or else the first few requests would just fail until the instance spun up to handle the request.

Yeah definitely. The dangerous part is having migrations or some other preparatory task be a part of it. When you're mixing concerns like that, you open yourself up to more complex failure cases and a larger surface area for bugs to arise.

I feel that when you've grown to the size where zero downtime deploys are so important you'd do custom code just to enable them, it would be best to just invest in industry best practices and do backwards compatible deploys.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#114

Post by chipdart »

dewey wrote: post #110
chipdart wrote: post #100

Earlier quoted context omitted.

> That's just a small part of Kamal (...) That does not address OP's point. Any plain old reverse proxy handles this usecase, particularly those already used extensively as ingress controllers for container orchestration systems. Some even support specialized routing strategies. Other than NIH, what exactly does kamal-proxy offers that any other reverse proxy doesn't already provide?

Ah, I was more referring to why this tool was released and wanted to say it’s a component of their Kamal deployment tool and not so important on its own. There’s some more explanation on why they decided to switch away from Traefik: https://github.com/basecamp/kamal/pull/940

> There’s some more explanation on why they decided to switch away from Traefik:

Unfortunately no. I posted my comment after reading that PR. It says nothing of substance, and presents no justification. The case to roll out their own is summarized in this claim:

> No need for cordfiles, which were a hack used to stop Traefik from sending request to a container to make it unhealthy. Kamal-proxy just sends traffic where we tell it to.

The thing is, Traefik also "just sends traffic where we tell it to." It's one of it's most basic feature. In fact, that's the whole point of a reverse proxy. Why roll out their own?

And even if for any reason at all Traefik didn't cut it, why not pick one of the many many production-grade reverse proxies around? Why roll their own?

Those questions have no answer.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#115

Post by martinald »

miki123211 wrote: post #47
martinald wrote: post #9

Earlier quoted context omitted.

I don't think that's the real point. The real point is that 'big 3' cloud providers are so overpriced that you could run hugely over provisioned infra 24/7 for your load (to cope with any spikes) and still save a fortune. The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is. Slow single thread performance because of using the most parallel CPUs p…

You can always buy some servers to handle your base load, and then get extra cloud instances when needed. If you're running an ecommerce store for example, you could buy some extra capacity from AWS for Christmas and Black Friday, and rely on your own servers exclusively for the rest of the year.

But the ridiculous egress costs of the big clouds really reduce the feasibility of this. If you have some 'bare metal' boxes in the same city as your cloud instances you are going to be absolutely clobbered with the cost of database traffic from your additional AWS/azure/whatever boxes.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#117

Post by Nextgrid »

chipdart wrote: post #107
sgarland wrote: post #60

Earlier quoted context omitted.

> The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is. This. Mostly disk latency, for me. People who have only ever known DBaaS have no idea how absurdly fast they can be when you don’t have compute and disk split by network hops, and your disks are NVMe. Of course, it doesn’t matter, because the 10x latency hit is overshadowed by the miasma of…

> Of course, it doesn’t matter, because the 10x latency hit is overshadowed by the miasma of everything else in a modern stack. This. Those complaining about performance seem to come from people who are not be aware of latency numbers. Sure, the latency from reading data from a local drive can be lower than 1ms, whereas in block storage services like AWS EBS it can take more than 10ms. An order of magnitude slower. G…

It's still a problem if you need to do multiple sequential IO requests that depend on each other (example: read index to find a record, then read the actual record) and thus can't be parallelized. These batches of IO sometimes must themselves be sequential and can't be parallelized either, and suddenly this is bottlenecking the total throughput of your system.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#118

Post by Nextgrid »

martinald wrote: post #115
miki123211 wrote: post #47

Earlier quoted context omitted.

You can always buy some servers to handle your base load, and then get extra cloud instances when needed. If you're running an ecommerce store for example, you could buy some extra capacity from AWS for Christmas and Black Friday, and rely on your own servers exclusively for the rest of the year.

But the ridiculous egress costs of the big clouds really reduce the feasibility of this. If you have some 'bare metal' boxes in the same city as your cloud instances you are going to be absolutely clobbered with the cost of database traffic from your additional AWS/azure/whatever boxes.

Is database traffic really all that significant in this scenario? I'd expect the bulk of the cost to be the end-user traffic (serving web pages to clients) with database/other traffic to your existing infra a relatively minor line-item?

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#119

Post by sgarland »

chipdart wrote: post #107
sgarland wrote: post #60

Earlier quoted context omitted.

> The other thing is that cloud hardware is generally very very slow and many engineers don't seem to appreciate how bad it is. This. Mostly disk latency, for me. People who have only ever known DBaaS have no idea how absurdly fast they can be when you don’t have compute and disk split by network hops, and your disks are NVMe. Of course, it doesn’t matter, because the 10x latency hit is overshadowed by the miasma of…

> Of course, it doesn’t matter, because the 10x latency hit is overshadowed by the miasma of everything else in a modern stack. This. Those complaining about performance seem to come from people who are not be aware of latency numbers. Sure, the latency from reading data from a local drive can be lower than 1ms, whereas in block storage services like AWS EBS it can take more than 10ms. An order of magnitude slower. G…

While network latency may overshadow that of a single query, many apps have many such queries to accomplish one action, and it can start to add up.

I was referring more to how it's extremely rare to have a stack as simple as request --> LB --> app --> DB. Instead, the app almost always a micro service, even when it wasn't warranted, and each service is still making calls to DBs. Many of the services depend on other services, so there's no parallelization there. Then there's the caching layer stuck between service --> DB, because by and large RDBMS isn't understood or managed well, so the fix is to just throw Redis between them.

Re: Kamal Proxy – A minimal HTTP proxy for zero-downtime deployments

#120

Post by chipdart »

sgarland wrote: post #119
chipdart wrote: post #107

Earlier quoted context omitted.

> Of course, it doesn’t matter, because the 10x latency hit is overshadowed by the miasma of everything else in a modern stack. This. Those complaining about performance seem to come from people who are not be aware of latency numbers. Sure, the latency from reading data from a local drive can be lower than 1ms, whereas in block storage services like AWS EBS it can take more than 10ms. An order of magnitude slower. G…

While network latency may overshadow that of a single query, many apps have many such queries to accomplish one action, and it can start to add up. I was referring more to how it's extremely rare to have a stack as simple as request --> LB --> app --> DB. Instead, the app almost always a micro service, even when it wasn't warranted, and each service is still making calls to DBs. Many of the services depend on other s…

> While network latency may overshadow that of a single query, many apps have many such queries to accomplish one action, and it can start to add up.

I don't think this is a good argument. Even though disk latencies can add up, unless you're doing IO-heavy operations that should really be async calls, they are always a few orders of magnitude smaller than the whole response times.

The hypothetical gains you get from getting rid of 100% of your IO latencies tops off at a couple of dozen milliseconds. In platform-as-a-service offerings such as AWS' DynamoDB or Azure's CosmosDB, which involve a few network calls, an index query normally takes between 10 and 20ms. You barely get above single-digit performance gains if you lower risk latencies down to zero.

In relative terms, if you are operating an app where single-millisecond deltas in latencies are relevant, you get far greater decreases in response times by doing regional and edge deployments than switching to bare metal. Forget about doing regional deployments by running your hardware in-house.

There are many reason why talks about performance needs to start by getting performance numbers and figuring out bottlenecks.

Post reply on HN