So I explained it to Claude and made it write a Python script where I could manually set a few fixed times and it would adjust the SRT file, and it worked perfectly.
I literally paused the film and did that in under 5 minutes. It was amazing.
So fixing a lot of small things has become easier at least.