Outside of the PQ algorithms not being as thoroughly vetted as others, is there any negatives to shifting algorithms? Like even if someone were to prove that quantum computing is a dud, is there any reason why we shouldn't be using this stuff anyway?
Practically though, there are some downsides. Elliptic curves tend to have smaller ciphertexts/keys/signatures/so are better on bandwidth. If you do everything right with elliptic curves, we're also more confident in the hardness of the underlying problems (cf "generic group lower bounds", and other extensions of this model).
The new algorithms tend to be easier to implement (important, as a big source of practical insecurity is implementation issues. historically much more than the underlying assumption breaking). This isn't uniformly, e.g. I still think that the FN-DSA algorithm will have issues of this type, but ML-DSA and ML-KEM are fine. They're also easier to "specify", meaning it is much harder to accidentally choose a "weak" instance of them (in several senses. the "weak curve" attacks are not really possible. there isn't really a way to hide a NOBUS backdoor like there was for DUAL_EC_DRBG). They also tend to be faster.