31–40 of 41 posts
Seems like the kind of thing that a Deployment should be able to manage on its own... some kind of DrainPolicy object maybe? Also, if the previous ReplicaSet a Deployment is rolling past has several pods, maybe only some of them need to stay alive (maybe some drain sooner than others.) Perhaps the whole endeavor should just be to make Pod drainage a bit more explicit than just terminationGracePeriodSeconds... perhaps…
Earlier quoted context omitted.
yeah I don't know why terminationGracePeriodSeconds hacks didn't work. It could have been a different, unrelated factor that we didn't discover. It certainly could have been service-loadbalancer/haproxy's fault instead of the termination grace period itself. I'm certainly happy to be proven wrong there.
Not 100% sure about your scenario, but if you set a preStop hook to an exec probe you can arbitrarily delay shutdown inside the gracePeriod, because the kubelet won’t terminate the container until preStop returns. So if you set a 5 hour grace period, and a preStop hook that invokes a script that doesn’t return until all connections are closed (but which tells the container process not to accept new ones) you can cont…
One extra thing I remember that was sort of problematic was that when a pod was Terminating it'd get removed from the Endpoints, so any tooling that was using the API info to keep an eye on connections was basically unusable at that point.
> We still have one unsolved issue with this deployment strategy: how to clean up the old deployments when they’re no longer serving (much) traffic. Could probably solve this with a readiness probe / health check of sorts that is smart enough to know what low usage means.
So... just a blue/green deployment with a 24h delay before deleting the old cluster?
Live demo: https://youtu.be/1kjgwXP_N7A?t=10m46s Code: https://github.com/kstmp/metacontroller
Am I missing something, or wouldn’t it be as “simple” as connecting to the running container and running netstat and conditionally killing the pod based on the number of connections? I bet you thought of that, so I’m curious why it didn’t work for you.
It feels like it should be possible to fix the reconnect experience, especially in the planned termination of underlying container case: if you ask the client to reconnect, rather than abruptly disconnecting them then they could possibly even wait until their new session was fully established before dropping the old one. That doesn't take away from my appreciation of the pattern, though: I'm very much in favour of ro…
We actually have this functionality as well: we can send a signal to the process which will cause it to display a message to the user asking them to reload to upgrade. There is a similar feature in slack and riot that I've seen as well. Making this handoff automatic is definitely possible as well, though we do want people to reload occasionally to get new client-side code.
I haven't checked to see if autobahn has this functionality.