I disagree.
Designers are best at finding attractive looks. User Interface design (aka, usability) was a computer science specialty long long before it was a thing people with degrees that have anything to do with photoshop or color have been doing it.
I constantly see designers put out things that are non-standard and unintuitive (I work in the iPhone world). I don't want to hear any "no-true scotsman" about that how they're bad designers, they're not. "Design" people generally come from information display background, not interactivity design background. Interactivity was a discipline that came from ergo and HCI researchers, not design schools.
But usability is different than information display. They throw expensive, non-functional items out there all the time, as well as ridiculous control schemes that work like crap.
For this reason alone, ignoring designers, and at best working with a usability expert, could be productive, but worrying about people who specialize in visual appeal and just try to stick onto interaction design because it sounds good on a resume is a bit rich.
UX as a field sucks because too many people who were taught to make pretty things are putting on airs as if they have an education on making things that work well. Actual usability people (with training and all that) are gems usually. If you don't know how to do a usability study, with controls and all that, you're probably missing a few things to prove yourself as a user interaction designer.