> I would envision it to pretty much allow to listen on raw keyboard and mouse events.
This is pretty much how the BRIEF editor for programmers worked, which was used in MS-DOS around 1990. In my opinion this was one of the best programs that I have ever encountered and I have never encountered again some of the good UI features that existed in the BRIEF text editor and in the XTree file manager, both for MS-DOS.
BRIEF had a tiling-window user interface with optional menus and it came with a default behavior, but it could be customized extremely. For each kind of key press or mouse event you could bind a function to be executed, which replaced the default behavior. For instance, the default behavior for pressing most keys was to invoke a function "self_insert", which inserted in the current text buffer and the current cursor position the character corresponding to the pressed key, but it was easy to change the behavior to anything else, e.g. to do auto-completion or inserting some program template.
The functions bound to events were written in a custom language similar to LISP, so in this respect BRIEF resembled Emacs. A great number of text manipulation functions were provided, so it was easy to describe complex actions in the event handling functions.
This was what I liked most at BRIEF, that by rewriting all the event handling functions it was very easy to convert it in an editor that no longer resembled the default BRIEF editor at all, but which behaved in whatever way you believed that an editor for program sources should behave.
So the text editor provided an extensive library of functions for handling text, text buffers, windows and files and a LISP-like language for gluing all these into a text editor. It also had a default implementation of a text editor, to be able to use it immediately, but that could be replaced partially or totally, when you had more experience and you knew what kind of behavior you want.
I assume that Emacs can be customized in a similar manner with BRIEF, but when I looked at Emacs it required a much greater initial effort than BRIEF, to reach a stage where you are able to modify it.
The default editing mode of BRIEF was very intuitive, at least for someone familiar with the IBM PC keyboard, so you could use it easily out of the box, before learning to customize it. Then, like most MS-DOS programs of that time, it came with excellent documentation, so it was very easy and fast to learn how to rewrite its event handling functions. With Emacs, both learning how to just edit texts with it, and then how to modify it according to what you prefer, is much steeper and more time consuming than it was with BRIEF.