I don't disagree with anything you're saying
except that Termux is, of and by itself, a sufficient replacement for a full desktop system.
I'd tried to make that happen over the course of 2.5 years, a full year of which was with Termux. And as mentioned: the best damned thing to happen to me on Android.
Termux is really nice, and for the 700 - 800 or so packages it has, there's a tremendous amount of utility. But you're still stuck with limitations.
1. The Termux-dies-randomly problem is, AFAICT, inherent to Android and its memory management model. Unless Termux can somehow indicate that it is not a terminable process, that's not going to go away.
2. The terminal-performance issue appears to be specific to Termux, as other full-screen console tools (ConnectBot SSH client) do not have this problem. Since it manifests on both local and remote connections w/ Termux (e.g., scrolling backwards within a manpage on a remote system), it seems to be the local terminal code on Termux itself. Of all my complaints, this seems the one most readily fixed. But it has not been. (I have reported it. Several other issues I've noted have been rapidly addressed.)
3. The SSH lag and hang issues I saw seemed to cover all Android SSH clients, and were either Android, its networking stack, or quite possibly my Very Shitty WiFi Network (since somewhat improved). But in all Not Particularly Heartening. Wireless sessions from a laptop didn't have the same lag issues.
4. Package selection. Unless you switch to a full Android/Debian system or something like it, you're limited in what's available. Commendable effort, and a Hard Problem. Just ... well, it leaves me short.
5. The Android storage model. Termux is Just Another Android app, and has Very Limited Access to storage. Which fucking sucks as there's a 128 GB MicroSD card on the device, but almost completely unusable. Rooting your Android might get around this, but if you're stuck with an unrooted (or unrootable) device, Thou Remainst Fucked.
6. Shell only. There's no X.org or other graphical interaction. (Other than Android native, which leaves you with that platform's bastardised and pitiful apps selection, on which I've written elsewhere.)
Mind: I really wanted to like Termux. I kind of wanted to like Android. I have a Samsung Tab A and Linksys Bluetooth folio case/keyboard. In terms of a portable form factor, it's really fairly close to my ideal. But in terms of capabilities, performance, behaviour, applications, surveillance, and vendor shitfuckingheadedness, from Samsung, Google, and Linksys, and absolute and complete clusterfuck. I want to run as far as I possibly can from any of them. I absolutely recommend AGAINST any of these vendors.
The simple truth is that I want and need something that they are unable, or far more likely, unwilling to provide. Neither my financial nor time budget for exploring alternatives is large, and the profound level of frustration I've had in this (and earlier) Android excursions has left some keep and I'm sure long-lasting scars.
I've tried and used many other systems, few of which strike me as desireable. I don't care for Microsoft's products (though the MS Surface tablet, again, strikes me as well-considered hardware). Apple's tablets and laptops are shiny, but ultimately annoying. I'm typing this on a retina iMac, which has a luscious display but an annoying OS. I've been partial to Thinkpads for nearly two decades, but find Lenovo's recent shenanigans massively off-putting. I'm also no longer generally satisfied with the traditional clamshell laptop design, yet don't see what might yet supplant it. Portrait reading for documents is a very compelling feature, but what I see are either tablets (with no or poorly-considered keyboard and mouse options), or hybrid devices with hinging mechanisms that leave me in a cold sweat. (Maybe they Really Do Just Work and Will Prove Rugged in Actual Use, but I Have My Doubts.)