My terminal responds more quickly than it did in 1983. In every other way computers have subsumed televisions as entertainment devices, and have performance characteristics appropriate to their main uses.
Because of Moore's law advances, the industry has learned to design for the next generation CPU. Before Moore's law was understood, industry designed for the current hardware and everything was extremely fast.
In a way I wish RIM had won the smartphone wars, but after the patent loss and removal of the click wheel, the devices were far less usable.
Apple's foray into Skeuomorphism offers us a history of the competitive landscape for usability and design. Skeuomorphism is a way of avoiding the learning curve associated with abstractions. If the software is meant to let the user take notes, making the screen yellow with lines on it helps the most abstraction-resistant users grok what is going on.
Most adults can't do basic algebra, so it should be no surprise that the dominant mobile software platform succeeded only after substantially dumbing down the UI to the point where it was essentially abstraction free.
With Jony Ive taking more control of iOS we are finally breaking free of the Skeuomorphic training wheels. Google's material design is an ambitious attempt to create an abstraction that retains some of the more textural and familiar "material-ness" of real-world objects, without resorting to crude mimicry. I wouldn't say it's close to perfect, but it is promising.
So consider the need for a fast GPU and high res graphics in a mobile device to be (in a sense) a tax that we must pay to help the small percentage of abstraction-phobic luddites understand how to write a note or add a meeting to the calendar. It also makes the device useful for gaming and for watching TV, which many people enjoy doing and which create massive revenue for content companies.
Apple came to dominate the smartphone market because it understood that consumers wanted an all-purpose device that didn't shove too many abstractions down anyone's throat, had decent video games, would let you watch TV and movies, did not feel like using a computer.
With Android, Google has mostly stayed behind by about one year to cut costs and capture market share, but has increasingly been focusing on delivering a top-tier experience and offering/supporting top-tier hardware.
Movies, shows, and songs make up 99% of most users' stored data. So of course the device needs to work with that data. Any comparison to 1983 cannot consider such massive data because it was unfathomable. Back then we thought calendars, note taking, simple spreadsheets, etc., were what computers were good for. Nobody realized that they would become handheld TVs with a built-in gossip rag with custom content about all of our friends.
We will someday reach a point where a mobile device reaches "peak dopamine", meaning that no further improvements will be necessary to make the device more entertaining or more addictive. We have a long way to go, and a lot more transistors will be needed in the hardware to reach peak dopamine, so we can expect the trend to continue. UI responsiveness figures in somewhere, but is obviously not the main driver of hardware cost increases.