Earlier quoted context omitted.
I used to feel the same way. Then, I gave away my Apple TV and bought a Chromecast. Searching for content.. jumping between content sources.. it's all easier on the phone.
Did you try Apple's Remote app with Apple TV? Is it still worse than Chromecast?
What I like about Chromecast is that their integration is simply a small button added to other apps that I'm already familiar with.
For example, I use Netflix as normal but basically choose where the output should go. It's actually quite similar to Apple's AirPlay integration although the Chromecast has never had an issue (AirPlay always seemed to have latency issues).