Exploring the Fragmentation of Wayland, an xdotool adventure

Windows gets to completely rearchitecture their compositor because they only provide one stable ABI to get pixels on the screen: link to USER32.DLL, create the necessary objects to represent a window of your application's class, then create and pump a message queue for it. It's ancient, but it works, and more specifically will never change. Even the higher level toolkits Windows ships ultimately are creating USER windows, and USER has been the only UI ABI since version 1.

macOS is the same way, except Carbon (a light modification to the procedural Toolbox API) and Cocoa (the Mac's first OOP toolkit) were "toll-free bridged" to each other rather than, say, writing Cocoa in terms of Carbon.

In contrast, X11 is a protocol anyone can implement and speak. There is no blessed library that you must use. No, Xlib doesn't count. Servers have to take their clients as they come. And Wayland, while very much deliberately stripped down from X, still retains this property of "the demarc point is a protocol" while every proprietary OS (and Android) went with "the demarc point is a library".

I think the hard part about the Linux desktop ecosystem and its development pattern is the cobbled-up-parts nature of the system, where different teams and individuals work on different subsystems with no higher leadership directing how all of these parts should be assembled to create a cohesive whole. We have a situation where GUI applications depended on X.org, yet the X.org developers didn't want to work on X.org any more. If the desktop Linux ecosystem were more like FreeBSD in the sense that FreeBSD has control over both the kernel and its bundled userland, there'd be a clearer transition away from X.org since X.org would have been owned by the overall Linux project. However, that's not how development in the Linux ecosystem works, and what we ended up with is a very messy, dragged-out transition from X to Wayland, complete with competing compositors.

Bazaar-style development seems to work for command-line tools, but I don't think it works well for a coherent desktop experience. We've had so much fragmentation, from KDE/Qt vs GNOME/GTK, to now X11 vs Wayland. Even X11 itself didn't come from the bazaar, but rather from MIT, DEC, and IBM (https://en.wikipedia.org/wiki/X_Window_System).

In my opinion, three basic things are needed:

- Device emulation: uinput covers this; requiring root is reasonable for what it does.

- Input injection. Like XTEST, but ideally with permissions and more event types (i.e. tablet and touch events.) libei is close but I think it should be a Wayland protocol.

- UI automation: Right now I think the closest you can get is with AT-SPI2, for apps that support it. This should also be a Wayland protocol.

None of these are actually easy if you want to make a good API. (XTEST is a convenient API, but not a particularly good one. Win32 has better input emulation and UI automation features IMO.)

Also the tangent about how crazy the compatibility layers are is weird. Yes, funny things are being done for the sake of compatibility. XWaylandVideoBridge is another example, but screen sharing is an area where Wayland is arguably better (despite what NVIDIA has to say) because you can get zero copy window and screen contents through PipeWire thanks to dmabufs.

Some of the lack of progress comes down to disagreements. libei mainly exists, by my best estimate, because the GNOME folks don't like putting things in Mutter, and don't want to figure out how to deal with moving things out of process while keeping them in protocol. (Nevermind the fact that this still has to go through Mutter eventually, since it is the guy sending the events anyways...) However, as far as I know, lack of progress on UI automation and accessibility entirely comes down to funding. It's easy to say "why not just add SetCursorPos(x, y)" and laugh it off, but attacking these problems is really quite complex. There was Newton for the UI automation part, but unfortunately we haven't heard anything since 2024 AFAIK, and nobody else has stepped up.

https://blogs.gnome.org/a11y/2023/10/27/a-new-accessibility-...

Color management is the perfect example of how a simple ask can be complicated. How hard could it really be? Well, see for yourself.

https://gitlab.freedesktop.org/wayland/wayland-protocols/-/m...

If Wayland lasts as long as X11 did, it's preposterous to not spend the time to try to get the "new" version of these things right even if it is painful in the meantime.

After all, it isn't like UI automation on Linux was ever particularly good. Anyone who has ever used AutoHotkey could've told you that.

There's so much hot debate about how bad Wayland is, how incorrect it is. But theres something I respect enormously about Wayland which is that: it is so so so much less than Xorg.

It uses the kernel's graphics buffers. It uses the kernel's mode setting. These alone are humongous differeniatiors.

There's so many other amazing glorious ways that Wayland is less. The protocol-centricity is vastly under rated, a massive win for the bazaar that can keep seeking truth versus the (imo utterly pathetic clining) absolutionist monolith style.

It's revolting to see such persistent bitter angry low user disdain, anger. Without any acknowledgement at all. That protocols allowing multiple implementations allows constant honing in, allows for dynamic change and evolution.

Reflecting on the Hindu Trimurti, a cycle of creation/newness, stasis/pattern, and decay & rot, it's amazing how the protest no-change/stasis-only voice has such a loud undying protest going. X is never getting better, has no room to improve, cursed by its own egocentric insanity which it has recursed into far far too far: which the core devs all agree.

It's not pleasant for everyone that Wayland allows a freedom of implementation. But generally most of the protest here has fallen away: support for major features is just here, on most implementations. That competitors can compete, don't have to keep using the same base is hugely advantageous to humanity. But the protest no-change anger-only voice is so loud. Doesn't know doesn't care.

Humanity should respect systems where competition and improvement are possible. X was a single consigned fate, with no growth or improvement. The competition of Wayland is an incredible breath of fresh air, and the growth of protocol competition here is telling, to not necessarily the "everything just works and is great" desire path of the low tech-ig orant beggar class, but which has enable so much Bazaar democratic figuring shit out, that still shares the ideas while allowing innovation within, in a way that few projects have ever enabled before. We are in a magic age of so so much, such cooperative competitive improvement, and it's just so unspoken, so missed, amid the squeaky wheels offering no actual technical critiques, unable to reflect upon the different (much better) age of possibility the bazaar model has opened us into.

The post shows a common issue with Wayland. The protocol is there, but each compositor handles things a bit differently, so tools like xdotool end up running into gaps or inconsistent behavior.

Wayland is improving, but there is still a difference between what the spec supports and what developers can rely on across the ecosystem.

A good look at why automation on Wayland still feels rough for some users.