They have a bit of a learning curve, because if the switch legs aren't straightened out before being pushed in, there's a chance they can be irreversibly crumpled. It can be a hard lesson if the switches were expensive.
De-soldered switches are especially prone to this problem because the leftover solder on the legs makes them 1) more brittle, and 2) irregularly shaped, so switches like vintage blacks are kind of a no-go for hotswap.
Some have reported the hotswap sockets pop out when forcing switches in, which I personally haven't witnessed myself. The impact ranges from a trivial matter of needing to apply more solder to the pad, all the way up to the hot swap socket ripping off the pad itself (very difficult to repair).
Hotswap sockets are less secure than switches that are directly soldered on, which means that one typically will want to use them in combination with a switch plate. Switch plates, however, have the counter-intuitive effect of raising the pitch of your keyboard's sound profile. Most people don't realize this. By a lot of standards, a plate-less keyboard does sound better, and with hotswap all those keys will start looking slightly crooked fast.
These nitpicks are things most users will never notice, but people who have been in the hobby for a long time do notice them, and those folks happen to be very vocal and influential. Then there's also the fact that anyone who is willing to design a PCB is probably savvy enough to desolder without issues. These factors are why there appears to be a strong anti-hotswap movement.
Overall, I'd still recommend using hotswap.