I was recently exploring "an OS from scratch" architecture with ChatGPT. Apart from the central idea that I wanted it to be a capability OS, I also started thinking about how I'd make GUI apps and what layers would naturally make sense.
The cleanest design I came up with works like this:
1. At Layer_1 OS provides panes or windows to which apps can render whatever graphics using OS gpu lib (think Raylib, but maybe a bit simpler). This is good for apps that want performance, games, text editors, things like that - so this is maximum control.
2. Next Layer_2 would be based on the browser engine (which, to some degree, would have to be developer - not a full browser, but just enough to render more or less modern html/css with a DOM-aware JS engine). Then provide some XML schema that would automatically translate into HTML/CSS to render various UI-widgets (defined by XML/HTML elements). This would eventually suggest a JavaScript UI framework on top, but no a requirement. This layer is flexible enough that it uses widgets defined by the system, but allows custom CSS and JavaScript.
3. Layer_3, or more like Layer_2.5 would basically allow custom HTML/CSS without relying on system widgets and XML schema.
Obviously you can mix all three layers into one GUI too. Finally layers (2) and (3) would require some client/server architecture, but instead of shipping it like a webapp, I'd probably provide some JavaScript API to talk to the backend (which in this case is accessible via a socket of some sorts). Apps themselves could still be single binaries that include html/css/js assets which are provided at launch time to the system UI renderer, which then uses them to render the UI via a webview in said pane or window.
I think it's pretty slick and it only gets complicated gradually, but it also requires solid OS plumbing and machinery to make this process smooth. None of the modern systems are capable of it in the way I described it, so people end up shipping nodejs/react apps with webview inside gigabyte-binaries.