A file descriptor is a handle to a file, and anything you have an fd to
is a file. This file carries a vfs implementation, such as that of pidfd, a device driver, or disk storage. The kernel does not distinguish between these.
If not being able to write makes it not a file, then files stop existing when a disk is full, and means that /dev/zero and /dev/null are not files - despite being at the heart of the whole "everything is a file" paradigm.
Pidfd was not made to make processes behave like files - that is what /proc is - but to solve problems with process related syscalls and PIDs, which are flawed and racey. The solution to that was to make APIs that treat processes as files, which gives you the ability to poll it like a file.
A streaming socket is exactly like a normal file. You read, write and poll. The only thing that is special is how to create it, but that is a design decision, not a technical limitation - see the plan9 file based API for making TCP sockets, which is trivially implementable in Linux.
Domain sockets are a bit different because of their side channel and would require more ctl files, but Linux's API is 99% magic files and ioctls so this is not that weird.
ioctls are not themselves bad design. In fact, scoping kernel functionality onto file handles is a great design and why that's almost the entirety of the kernel (device driver calls dwarf syscalls). The problem is not the design itself, but the fact that ioctl was not originally meant for it and got overloaded through several design iterations. This is what happens when you do organic design through more than 3 decades.
If you start out by defining a way to do file-scoped syscalls - and file does and always will mean "an fd" to a kernel - then you wouldn't have that awkwardness. That is what plan9 did: Take the learnings, and implement them clean instead of on legacy.