A clear complexity cost difficulty in extending these libraries. This separation force libraries to become huge complex monoliths. In Julia the equivalent is done with absolutely hilariously tiny libraries. In fact they are so small that many Python guys exploring Julia decide to not explore further thinking most of the Julia ML libraries aren't done or have barely started.
They are just not accustomed to seeing libraries being that small. That is possible in Julia because it is all native Julia code which means interfacing with other Julia code works seamless and allows you to mix and match many small libraries very easily. You can reuse much more functionality which means individual libraries can be kept very small.
For PyTorch and TensorFlow e.g. activation functions have to be coded specifically into each library. In Julia these can just be reused for any library. Each ML library doesn't need to reimplement activation functions.
That is why you get these bloated monoliths. They have to reinvent the wheel over and over again. So yeah there is a cost which is constantly paid.
Every time you need to extend these libraries with some functionality you are paying a much higher price than when you do the same with Julia.