Rainbow Deployments with Kubernetes • Hacker News Board

Re: Rainbow Deployments with Kubernetes

#21

Post by bdimcheff » Wed, Feb 14, 2018, 8:34 PM UTC

ninkendo wrote: post #19

Seems like the kind of thing that a Deployment should be able to manage on its own... some kind of DrainPolicy object maybe? Also, if the previous ReplicaSet a Deployment is rolling past has several pods, maybe only some of them need to stay alive (maybe some drain sooner than others.) Perhaps the whole endeavor should just be to make Pod drainage a bit more explicit than just terminationGracePeriodSeconds... perhaps…

yeah I don't know why terminationGracePeriodSeconds hacks didn't work. It could have been a different, unrelated factor that we didn't discover. It certainly could have been service-loadbalancer/haproxy's fault instead of the termination grace period itself. I'm certainly happy to be proven wrong there.

Re: Rainbow Deployments with Kubernetes

#22

Post by bdimcheff » Wed, Feb 14, 2018, 8:37 PM UTC

_asummers wrote: post #20

> We still have one unsolved issue with this deployment strategy: how to clean up the old deployments when they’re no longer serving (much) traffic. Could probably solve this with a readiness probe / health check of sorts that is smart enough to know what low usage means.

Yeah I think if restartPolicy were changeable at runtime, we could simply have the pods exit once their connections are drained enough. If we were to exit using the current strategy, they'd just be restarted by kube.

Re: Rainbow Deployments with Kubernetes

#23

Post by erikrothoff » Wed, Feb 14, 2018, 8:46 PM UTC

This was really interesting. I'm thinking about moving to Kubernetes and have wondered how to gracefully deal with websocket connections.

I'm curious though, if the rollout was over a couple of hours for example, why would reconnections be that big of a problem? We host about 10,000+ websocket on a $20 VPS, and the Go server hosting it crashes from time to time. A surge of 10,000 reconnections instantly afterwards has never lasted for more than a minute or so, so why is it so bad? Moments of peak load aren't that big of a deal, or?

Re: Rainbow Deployments with Kubernetes

#24

Post by codegladiator » Wed, Feb 14, 2018, 8:52 PM UTC

jkarneges wrote: post #7

wjossey wrote: post #2

Question for the OP- I haven't ever worked on chat services, so this may not be reasonable. Would it be possible to use some other termination endpoint that sits in front of the service, that allows you to maintain persistent connections to the clients, but make for more transparent swaps of backend services? So, for example could you leverage nginx or haproxy as the "termination" point for the chat connection, with…

This is essentially how Pushpin ( http://pushpin.org ) works. It can hold a raw WebSocket connection open with a client, but it speaks HTTP to the backend server, and the backend can be restarted without the client noticing.

This is a nginx module for this functionality

https://nchan.io/

I have used it before. Super easy to setup. Even with kubernetes.

Re: Rainbow Deployments with Kubernetes

#25

Post by bdimcheff » Wed, Feb 14, 2018, 9:06 PM UTC

erikrothoff wrote: post #23

This was really interesting. I'm thinking about moving to Kubernetes and have wondered how to gracefully deal with websocket connections. I'm curious though, if the rollout was over a couple of hours for example, why would reconnections be that big of a problem? We host about 10,000+ websocket on a $20 VPS, and the Go server hosting it crashes from time to time. A surge of 10,000 reconnections instantly afterwards ha…

In our case it's not the websockets that are the problem, it's the XMPP connection that each websocket connection creates. Logging in thousands of users takes several minutes. While a user reconnects, any conversations that the users are having with their website visitors are disrupted.

Re: Rainbow Deployments with Kubernetes

#26

Post by rockostrich » Wed, Feb 14, 2018, 9:10 PM UTC

gouggoug wrote: post #9

I'm not sure what problem the author is solving. I might be misunderstanding something. The author points out that the issue with Blue/Green/AnyColors deploys is that they need 16 pods per color at all time (which in their case would end up being 128 pods) and 24/48 hours for each connection pool to drain. But how is using a SHA instead of a COLOR any different? Unless I am missing something, and, if running 128 pods…

I think their problem is that they need to keep old deploys alive for a long time because they have stateful long-running connections. So a blue/green deploy wouldn't work because their blue deploy has to stay around for at least a month after green is deployed. There's no difference between the SHA and COLOR, I think the choice to use a git hash is because it's the logical choice (instead of randomly choosing a color from a list).

Re: Rainbow Deployments with Kubernetes

#27

Post by markbnj » Wed, Feb 14, 2018, 9:10 PM UTC

erikrothoff wrote: post #23

This was really interesting. I'm thinking about moving to Kubernetes and have wondered how to gracefully deal with websocket connections. I'm curious though, if the rollout was over a couple of hours for example, why would reconnections be that big of a problem? We host about 10,000+ websocket on a $20 VPS, and the Go server hosting it crashes from time to time. A surge of 10,000 reconnections instantly afterwards ha…

(work with OP on the same team) Basically there are a lot of other things that happen when a websocket connection is established and we don't necessarily have the capacity to handle that volume in a complete reconnect scenario, especially if the system is already near the daily load peak. We have hopes that autoscaling some things in the future will make it possible to handle peaks like this more gracefully.

Re: Rainbow Deployments with Kubernetes

#28

Post by jkarneges » Wed, Feb 14, 2018, 9:15 PM UTC

codegladiator wrote: post #24

jkarneges wrote: post #7

Earlier quoted context omitted.

This is essentially how Pushpin ( http://pushpin.org ) works. It can hold a raw WebSocket connection open with a client, but it speaks HTTP to the backend server, and the backend can be restarted without the client noticing.

This is a nginx module for this functionality https://nchan.io/ I have used it before. Super easy to setup. Even with kubernetes.

Interesting, I didn't think Nchan would be able to drive a raw WebSocket session with the client, but upon closer look it seems like it might be possible using the auth and message forwarding hooks. Very cool.

Re: Rainbow Deployments with Kubernetes

#29

Post by smarterclayton » Wed, Feb 14, 2018, 9:29 PM UTC

bdimcheff wrote: post #21

ninkendo wrote: post #19

Seems like the kind of thing that a Deployment should be able to manage on its own... some kind of DrainPolicy object maybe? Also, if the previous ReplicaSet a Deployment is rolling past has several pods, maybe only some of them need to stay alive (maybe some drain sooner than others.) Perhaps the whole endeavor should just be to make Pod drainage a bit more explicit than just terminationGracePeriodSeconds... perhaps…

yeah I don't know why terminationGracePeriodSeconds hacks didn't work. It could have been a different, unrelated factor that we didn't discover. It certainly could have been service-loadbalancer/haproxy's fault instead of the termination grace period itself. I'm certainly happy to be proven wrong there.

Not 100% sure about your scenario, but if you set a preStop hook to an exec probe you can arbitrarily delay shutdown inside the gracePeriod, because the kubelet won’t terminate the container until preStop returns.

So if you set a 5 hour grace period, and a preStop hook that invokes a script that doesn’t return until all connections are closed (but which tells the container process not to accept new ones) you can control the drain rate.

There is some app level smarts required - to have new connections rejected and have any proxies rebalance you. Haproxy does this in most cases, but the service proxy won’t (in iptables mode).

If that’s not the behavior you’re seeing, please open a bug on Kube and assign me (this is something I maintain)

Re: Rainbow Deployments with Kubernetes

#30

Post by bdimcheff » Wed, Feb 14, 2018, 9:52 PM UTC

One thing I didn't put in here that's also turned out to be useful: We can prerelease things relatively easily this way too. Each deployment has a git sha, and we can have a canary/beta/dogfood version that points at an entirely different sha.