So don't set the token to expire after the "heat death of the universe", make the user reauthenticate after an appropriate time for the service being used.
The problem is that you don't have control over what third parties are doing here. You may have control over the policies associated with tokens issued by your identity provider, but how do you audit the policies that third party services are applying to their tokens?
Can you give a more fleshed out example, why should you audit third party services?