Rainbow Deployments with Kubernetes
brandon.dimcheff.com
Rainbow Deployments with Kubernetes
1–10 of 41 posts
Re: Rainbow Deployments with Kubernetes
#2Question for the OP-
I haven't ever worked on chat services, so this may not be reasonable. Would it be possible to use some other termination endpoint that sits in front of the service, that allows you to maintain persistent connections to the clients, but make for more transparent swaps of backend services?
So, for example could you leverage nginx or haproxy as the "termination" point for the chat connection, with those proxying back to the kubernetes service endpoint, which then proxy back to the real backend service. So, when you go to swap out the backend service, nginx / haproxy start forwarding to the new service transparently, while still maintaining the long-lived connection with the client.
If this was doable, it would mean you'd only have to drain if you needed to swapout the proxy layer, which is likely a less-frequent task, and thus allows you more agility with your backend services.
Re: Rainbow Deployments with Kubernetes
#3Question for the OP- I haven't ever worked on chat services, so this may not be reasonable. Would it be possible to use some other termination endpoint that sits in front of the service, that allows you to maintain persistent connections to the clients, but make for more transparent swaps of backend services? So, for example could you leverage nginx or haproxy as the "termination" point for the chat connection, with…
It sounds like the product is already a proxy (Websocket -> XMPP) so I'm not sure what exactly they're deploying multiple times per day.
It also seems like they could have done simple blue/green and extend their websocket protocol and client to support a hand-off "hey, there's a new version of the proxy, reconnect in x seconds" message (and have idempotency of messages) such that they could have a rather smooth schedule where everyone can reconnect then disconnect gradually over some period of minutes instead of hours and not have either sudden spikes or any period of interruption for clients.
Re: Rainbow Deployments with Kubernetes
#4Question for the OP- I haven't ever worked on chat services, so this may not be reasonable. Would it be possible to use some other termination endpoint that sits in front of the service, that allows you to maintain persistent connections to the clients, but make for more transparent swaps of backend services? So, for example could you leverage nginx or haproxy as the "termination" point for the chat connection, with…
It sounds like the product is already a proxy (Websocket -> XMPP) so I'm not sure what exactly they're deploying multiple times per day. It also seems like they could have done simple blue/green and extend their websocket protocol and client to support a hand-off "hey, there's a new version of the proxy, reconnect in x seconds" message (and have idempotency of messages) such that they could have a rather smooth sched…
Or just a generic reconnect in case the websocket connection breaks for any reason really.
Re: Rainbow Deployments with Kubernetes
#5Question for the OP- I haven't ever worked on chat services, so this may not be reasonable. Would it be possible to use some other termination endpoint that sits in front of the service, that allows you to maintain persistent connections to the clients, but make for more transparent swaps of backend services? So, for example could you leverage nginx or haproxy as the "termination" point for the chat connection, with…
In practice you probably would want pretty thin proxy layer as you said, which is then forwarding requests to other services as needed...but you would still need to re-deploy this proxy layer, and would thus need a similar solution to the one proposed in the article.
Re: Rainbow Deployments with Kubernetes
#6Curious about the 24h-48h burndown...could it potentially be longer for you guys or is there some mechanism in place to force disconnection (and thus risk a spike) after some TTL?
Re: Rainbow Deployments with Kubernetes
#7Question for the OP- I haven't ever worked on chat services, so this may not be reasonable. Would it be possible to use some other termination endpoint that sits in front of the service, that allows you to maintain persistent connections to the clients, but make for more transparent swaps of backend services? So, for example could you leverage nginx or haproxy as the "termination" point for the chat connection, with…
This is essentially how Pushpin (http://pushpin.org) works. It can hold a raw WebSocket connection open with a client, but it speaks HTTP to the backend server, and the backend can be restarted without the client noticing.
Re: Rainbow Deployments with Kubernetes
#8It feels like it should be possible to fix the reconnect experience, especially in the planned termination of underlying container case: if you ask the client to reconnect, rather than abruptly disconnecting them then they could possibly even wait until their new session was fully established before dropping the old one.
That doesn't take away from my appreciation of the pattern, though: I'm very much in favour of rolling releases forwards, rather than being limited to two colours.
Re: Rainbow Deployments with Kubernetes
#9I'm not sure what problem the author is solving. I might be misunderstanding something.
The author points out that the issue with Blue/Green/AnyColors deploys is that they need 16 pods per color at all time (which in their case would end up being 128 pods) and 24/48 hours for each connection pool to drain.
But how is using a SHA instead of a COLOR any different? Unless I am missing something, and, if running 128 pods and 24/48 hours of draining is the issue, then using SHA instead of colors is not solving those 2 issues.
You'll still need 16 pods and 24/48 hours per SHA-deploy, and you're actually making it worst by not using fixed colors since you have quite a lot more SHA at your disposition.
Re: Rainbow Deployments with Kubernetes
#10Earlier quoted context omitted.
It sounds like the product is already a proxy (Websocket -> XMPP) so I'm not sure what exactly they're deploying multiple times per day. It also seems like they could have done simple blue/green and extend their websocket protocol and client to support a hand-off "hey, there's a new version of the proxy, reconnect in x seconds" message (and have idempotency of messages) such that they could have a rather smooth sched…
Or just a generic reconnect in case the websocket connection breaks for any reason really.
That would require an interruption in service between the server closing the old connection and the client reestablishing a new one, which they sought to avoid.