This is what I mean by setting
vengeance up vs. "the bare mininal extent required to protect others".
I agree that there are still people who needs to be locked up, including under possibly brutal sentencing terms. E.g. Anders Behring Breivik who carried out the 2011 terrorist attacks in Norway is up on a parole hearing today - if he gets out (it'd take a miracle) it doesn't matter if it's free will (in any sense of the term) or purely deterministic processes that leads him to act if he is still probabilistically likely to carry out another attack - it's still more important to keep the public safe from him than to worry about whether or not he had a genuine choice.
To the extent there's a measurable effect of seeing the punishment of others, that too might be a justification, though I find that dicier. But if that is done based on a cold calculation of how it affects offending rates that to me is still very different from vengeance.
So to be clear I'm not arguing it removes the justifications for punishments, and I agree with your mention of seeing it as a game theoretic situation - even really harsh punishments might be the best choice in some circumstances. But it removes "they chose this themselves" or "they deserve it" as a justification (the feeling is still valid; acting on it becomes cruel), and that makes for a very different legal system. It also shifts a focus towards considering that if it's not about vengeance, then you should look at how to actually optimise the balance between the public good and treating the prisoners as humanely as you can without reducing the public benefit of the punishment too much.
Norway is far from achieving that, but if you look at e.g. Bastøy fengsel [1] ([2] has links to assorted longer articles), you can get one idea of what de-emphasising vengeance and taking rehabilitation and trying to address reoffending rates might start to look like (though Norway has by no means solved that, Bastøy also certainly haven't made it worse, but it also is not the norm in Norway either - yet anyway). For some people, it might well be that harsher prisons works better, or are necessary (I doubt Breivik will ever get the chance to serve his time somewhere like Bastøy for example). (Bastøy is the - so far - culmination of a process of reform that started in the 1960's, at a point where Norwegian prisons were quite brutal and the brutality of it eventually triggered a strong counter-reaction.)
For others even less prison (Norway as it is have sentences that would look comically short for e.g. Americans or British people). But evaluating it as a "game theoretic situation" as you say would in any case very likely lead to very major change to how the legal system would work most places in the world (including Norway)
[1] http://bastoyfengsel.no/English/
[2] https://en.wikipedia.org/wiki/Bast%C3%B8y_Prison