If I were building a terminal from scratch... Hm.
Attempts to improve the terminal are not backwards-compatible or do not otherwise provide a clean upgrade path. I would love to use one of those fancy graphical terms that comes up on HN every now and then, but because they only work with a special toolchain that speaks its language and its language only there's no way they'll ever gain critical mass and take off. Any attempt to improve the terminal has to be something that I could switch to on a whim and then gradually grow into.
I really like the "curious fact" from higher in this thread and I think that it's important. Whatever the solution is, I think that it'll necessarily involve separating "business" logic from presentation logic, and giving presentation logic to the terminal while things under the shell handle the actual work. Right now, one of the biggest problems with writing a CLI tool is that TUI work is a huge pain in the ass. It's harder to make a table in the terminal than it is to make a table on the web! That's insane! And that suggests a straightforward solution: Move display logic into the terminal so that it can handle layout and rendering. And you do that the same way that that's been done in literally every situation that's ever been done: You ship a small program that constructs a view that can take advantage of a more favorable execution environment.
So what does it mean to "ship a program that constructs a view"? Well, even if your program is written in a deeply standardized, weak language (e.g. ANSI escape codes) it's still a program. Formal languages theory, hooray. So we're already operating in this formalism, which makes our job much easier and the solution relatively obvious - Instead of inventing an entirely new paradigm from the ground up we just need to jam a more powerful language into an unused corner of the existing one.
This is where I go down the rabbit hole, because I don't know enough about the specifics of ANSI escape sequences. My goal, essentially speaking, is to find a set of sequences of control characters (a language) that will be ignored by everything except my New Terminal, map that set to some convenient programming language (ideally by a trivial "prefix-identity-suffix" construction), build a platform for building user interfaces that uses that programming language, and then build... something that augments existing programs with New Terminal front-ends? A lump of stuff into my shell RCs that detects New Terminal and augments the prompt strings with New Terminal apps that wrap the outputs of commands that look right?
Then the question is, what does my platform look like, and what convenient programming language am I using? I'd probably go with something like js or guile. Maybe just js so I can take advantage of extensive existing UI tooling. I mean, if you look at this sufficiently sideways, I've essentially duplicated the evolution of HTML+JS webapps - jam a new language (js script tags) into a corner of the existing language (html) that will be ignored by systems that don't understand the new thing and then implement a platform (DOM manipulation and such) that allows the creation of fancy user interfaces (webapps) using those tools. And really, I think that that's the clincher - that evolution happened when client machines became powerful enough that it was useful to move presentation logic out of the server and into the client, and that's exactly what we need to do with terminals.
(thought: interaction. callbacks return a sequence of characters that're typed into the terminal? so you can click on a table header and the client sorts and re-renders the table, or you can click on a button and to run a convenient follow-up invocation? ahhh, yeah - if the backend program (ls et al) is transient its interactions will render commands for follow-up operations, if it's persistent its interactions will render keystrokes for interacting with it - you'd click on the headers in top and it'd send `p`, `t`, etc. that's really clean.)