GIMP: My understanding is that at some point the UI design brahmins at GNOME decided that GTK+ must not support MDI (as in the UI paradigm where you have a container window with subwindows floating around inside it), and moreover the same kind of user interface should be implemented at the top level because a good window manager should be able of keeping an application's windows together. Well, no such window manager ever emerged, and as a result using GIMP is a perpetual search for what workspace a particular tool window disappeared to, or having to fight with it deciding that the minimum size of particular windows (tall toolbars, large viewports) is greater than the 1366x768 of my laptop screen. This is frustrating to the point of unusability.
Krita: Constant random crashes, like seemingly every complex Qt application.
Inkscape: I grew up on CorelDraw so maybe I just have the wrong set of expectations, but the UX is full of papercuts and inexplicable choices. For just one example that annoyed me recently, it seemed completely impossible to get snapping to grids and guidelines to behave; some objects would just refuse to snap to certain targets (different set for each object), and generally the snapping seemed to be based on arbitrary "basepoints" (only one per object; can you change those?) rather than bounding boxes.
(Blender, brought up elsewhere: actually quite good (modulo maybe the confusing refcounting/resource management), once you get on top of the learning curve. I hope 2.8, which I haven't tried yet, hasn't changed this.)