Emotional about X11: I'm creating a pure X11 “emoji keyboard”

Please let me rant a bit about X11 input APIs.

One of the major flaws in X11 is its poorly designed keyboard input system. When a key is pressed, the keypress event sends a "keycode" - an 8-bit number that references the current layout. This means you're limited to injecting characters that are present in the current layout.

The implications of this design are frustrating. For instance, if you're connecting to a remote system via VNC and the client and server have different keyboard layouts, you'll run into all sorts of issues. Similarly, how do you create an on-screen keyboard that can inject keypresses for characters not available in the current layout? And what if you want to programmatically send some text, but the user has the wrong layout active? It's a mess.

I guess because of this in Debian most of characters in on-screen keyboard didn't work (they "fixed" it by showing only characters present in active layouts instead of fixing the root issue).

The common workaround for this is an ugly hack: you modify the keyboard layout, find unused spots, add desired characters, send key press event and restore the keyboard layout back. See the code: [2]

Also, Wayland which was supposed to get rid of legacy problems, seems to have inherited this ugly design. Also, there seem to be no sane API for managing layouts or switching them programatically, or subscribing to layout change event. Also, you cannot use modifiers like Ctrl to switch layouts because then combinations like Ctrl + C stop working. Keyboard APIs on Linux are broken in the worst way possible since beginning, probably because most developers use only ASCII and do not have experience using multiple layouts.

A better idea would be to allow to send arbitrary Unicode strings and maybe integrate regular input and IME input (input system for typing Asian characters).

[1] https://www.x.org/releases/X11R7.6/doc/xproto/x11protocol.ht...

[2] https://github.com/Zirias/xmoji/blob/master/src/bin/xmoji/ke...

By that measure we'd still be using DOS these days (which was also productive and usable... and indeed the initial backlash against this nrefangled "Windows 95" thing kept going for a while, not very dissimilar to the X11 vs Wayland debates)

What the GP is complaining about isn't Unicode in general but Unicode Emoticons[1]. Characters can represent any thought, but emojis represent only what the committee agrees to, and the committee seems quite political.

What they should have done is to just have two characters and . And you could have text like:

    I'm not doing it pouting

If the renderer supports the "pouting" emoji it would replace the text and if it doesn't it would just render:

    I'm not doing it *pouting*

Everyone would be free to create emojis. You could pick your own emoji provider. So if the emoji doesn't exist locally it would be fetched from `http://emojiprovider.tld/pouting`. If you don't like it you can install another one.

It's ridiculous that there's no "pouting" emoji but there 6 emojis for pregnant men.

[1]: Yes, they're called emoticons in Unicode, not emojis. The term emoji entered English later.

For Wayland, keyboard input is "out of scope". (edit: Not entirely, just verified it does forward very basic events, but it's a thing wayland doesn't want to handle, while X11 originally had its own keyboard/mouse/etc drivers) What's typically used is XKB (inherited from X11) with a different backend ([edit: raw events]). So yes, in practice, you'll have the same broken design.

> Also, there seem to be no sane API for managing layouts or switching > them programatically

Layouts only exist as "data" as far as the X server is concerned, clients must fetch them and map themselves from the key codes. Libraries like xkbcommon (or even grandfather Xlib) do the job for you. That said, there are APIs to modify the mapping (by publishing messages/events) as you wish. The ugliness is, apart from the fact that you're forced to fiddle with the mapping at all, that you can't guarantee another client will process everything in sequence. It might apply a new mapping before processing all its queued key press events. That's why my code adds delays between fiddling with the mapping and sending the events.

Yes, it's extremely ugly. Still, at least for me, it works. Try it out ;)

And yes, Windows is doing better here, there's a Unicode-flavor of keyboard events available.

The only thing I can think of is that some clients might want raw keyboard events. It does seem to me (an outsider) like Keysyms are the better level of abstraction for input for the overwhelming majority cases, and the translation should happen as early as possible in the server, and that's even before ignoring things like the asynchronous nature of xlib making keyboard remapping &c. unreliable for sending key events outside of the primary keymap.