Here are my comments and concerns. 1. Minimal latency from keyboard to screen. 2. As others have mentioned here, awareness of line wrap, so that cut and paste work correctly at all times. Not just for raw output from a command. Editors too should communicate with the terminal to indicate that it is wrapping lines. 3. 24-bit color. 4. Apps (shells, editors) can query and set the window icon as well as the window title…
"cat" could never work of course, or the general idea of a terminal that has only a pair of input/output streams must be given up. I don't think people are willing to do that. A sensible protocol to draw raster graphics would be nice, though. (And it might exist, I don't even know). I don't care whether I need to type "cat" or the name of a terminal-based image viewer.
We have an idea in that direction, where each program in a pipeline can optionally specify a file type, and the last file type hint in the pipeline determines how the stdout is rendered. So if you have "cat image.jpg", then "cat" can indicate a file type of image/jpg, causing stdout to be interpreted as an image. But if you have "cat image.jpg | grep foo" (just making something up here), then grep can indicate a file type of application/octet-stream because it cannot guarantee what the file type is.
The tricky part, from what I can see, is explaining the pipeline topology to the terminal, so that it knows that "grep" comes after "cat", even though command parsing happens in the shell only. That's still a big unknown. A previous project in this area (TermKit) mandated that applications specify a "Content-Type" header (like in HTTP) in their stdout, but that would break most existing programs, so it's not an option for us.