I feel like Android 5 is a step back, UI-wise. One of those "change for the sake of change" things. I don't see how a triangle, square, and circle are more user friendly than a back button, a home button, and a tab button. Chrome is also crashing constantly on 5.03 and the new keyboard is very very bright at night. Not a telephony issue, but I feel like 5.03 was a step back and I've actually contemplated downgrading…
If we're still talking about a Nexus 4 or 5 here, it's Google who pushes the updates, not the carrier, as they are truly unlocked devices beholden only to Google. As such, you can download the previous versions of the OS all the way back to the initial release version[1]. The process to revert to an older version is not difficult (given the audience here) but if you've never done it before it can be a pain. The instructions are on the linked page.
For the Nexus 4, you can go all the way back to 4.2.2 (a nice, stable OS version if I may say so), and back to 4.4 for the Nexus 5.
If instead you are talking about another manufacturer like HTC or Motorola...good luck. Every manufacturer approaches downgrading/upgrading differently, not all manufacturers provide older ROM images, and some phones simply can't be downgraded once they've taken an OTA update.