Totally not a problem. If you are in the f-off frame of mind then probably for the best!

You won't be pushed that hard for an exit interview if it was pretty clearly not a good fit. Some of these transitions are a relief for all concerned (employee AND manager).

If a manager has someone with leadership / significant upside potential I do think a discussion is worth doing if possible.