The mathematics are less well researched, and at least for some algorithms the keys are quite large. (For instance, a pretty standard sized Niederreiter key and a single Niederreiter signature put the minimum size for a certificate over 1 MB, which is a lot less onerous than it used to be, but a bit big for doing a lot of copying on cellular networks.)
I haven't heard this expressed anywhere, but it could also be that the McEliece and Niederreiter algorithms have a bit of a feel to them like the Merkle-Hellman knapsack algorithm that was thought for a long time to be secure.
Though, if you're being paranoid and can spare the cycles, it wouldn't hurt to take Elliptic Curve Diffie-Hellman (or ECIES) exchange, encrypt that using RSA (or ElGamal), and then Encrypt that using Niederreiter (or McEliece), like a bunch of Russian dolls nested by key/message size. All three crypto systems would need to be broken in order to recover the key.