Yep. You lose sockopts (trivially restorable), async ownership (less trivially restorable, but with some kernel patches, it’s possible), and epoll registration (which omg, wtf.).
The other problem is that we found runtimes that would immediately send data after a non blocking connect (or receive data), and you needed a way to kick these off again. It was really a nightmare to restore all that state in a running socket.
The real solution was to add a new address family, and patch the kernel (as opposed to a module), that could dynamically switch back and forth (think AF_KCM). Unfortunately, I never got around to that.