Understanding the working of X11 from the ground up (2017)

Looking over this stuff, I get an uneasy feeling. X was a brilliant solution to problems 35 years ago, where you barely had enough memory for a full frame buffer and the graphics model was primitive (no antialiasing or transparency). From what I've seen, Wayland is a good solution for the model maybe 15 years ago, where you're willing to accept the frame-and-a-half latency hit from having a compositor in the path.

Basically, my question is whether anybody is working on a solution for the hardware we have today, where there are hardware overlays, the GPU throughput is orders of magnitude higher than what's needed to composite a desktop, but we still care about memory bandwidth, largely because all of resolution, bit depth, and frame rate is still going up. And you really want to be able to schedule every pixel that gets presented with a high degree of timing accuracy.

I've become familiar with Windows, but feel like it's a bunch of hacks. They will promote a window to an overlay, but it's all a heuristic, and I'm pretty sure latency will stutter while it does the transition in and out. And the new flip-model stuff actually regresses the ability to do smooth window resizing.

I'm not even sure macOS is working on lowering latency. I haven't seen any mention from them of hardware overlays.

Most of the attention these days seems to be in the VR/AR space, where latency is obviously important, and there's more of a green field with respect to legacy compatibility, but I'd love to see this problem space actually solved on the desktop. Doing this right is not easy - it probably requires some form of soft real time discipline across the graphics stack (including GPU scheduling) so you can race the beam.

I have a lot of replies to this, but the back-and-forth format that HN has isn't the best for collaborative discussion. Imagine us sitting in a room with a whiteboard, rather than bickering at each other at loud volume. If you're ever in south bay area and want to meet up at a coffee shop, let's :)

> Basically, my question is whether anybody is working on a solution for the hardware we have today, where there are hardware overlays, the GPU throughput is orders of magnitude higher than what's needed to composite a desktop

Hardware overlays are... not that popular on desktop devices? You just stomp on the gas pedal with bandwidth and power consumption on consumer desktop gaming GPUs. Besides a couple YUV planes and a cursor overlay, you don't see that much in the way of CRTC capabilities on desktop. All of that's happening in the mobile space, because of power consumption reasons, where you basically have an SNES PPU in your CRTC with multi-plane compositing.

I remain unconvinced that compositors need to add a frame and a half of latency. 16ms is a lot of time to render a frame, and composition isn't too difficult of a task. It just comes down to buffer handoff, which can be smoothed out over time. Similarly, the new flip-model isn't the thing that makes smooth resizing difficult, buffer allocation and resizing has always been a pain and racy on Windows/X11. This is one of the things that Wayland fixes -- it was actually the thing that caused Kristian to start Wayland forever ago.

Presentation and timing is something I think should be possible, but I don't think it makes sense for every app to have to handle it. Multi-monitor scenarios are what make global timing difficult, and having per-monitor surfaces is just a pain to deal with from an API standpoint; if we did the work to have per-monitor timing information and split surface rendering, nobody wants to go through the work to set up two devices and present different ones at the right time.

You might protest that all monitor timing is roughly the same these days, but someone hooking up their laptop to a projector, you get radically different timing.

In many respects, I think graphics is in a local minima right now: it works well enough and we understand it well enough that nobody's really interested in pushing us to the next thing. VR/AR research might, but that's been relatively unimportant as a target platform, commercially.

What's your view on the recent progress of Linux gaming?

> Win7/8

Windows 8 was a major improvement saddled with an untested UI that someone was using to make their political mark. The internals of Windows 8 and Windows 10 are a major, major step forward from 7.

It is, including the author of this post. Unlike other controversial things like systemd or pulseaudio, Wayland was started by major Xorg contributors. I've watched some of their talks and can see why they went down this route.

I've tired Wayland a few times and whenever I do, I give it an honest shot for a few weeks, but tend to run into issues (Sway being pretty buggy or oddities with old X apps or Steam, etc.) I've heard Sway has gotten a lot better and I should really try it out again; probably the next time I setup a laptop.

I do wonder what plans FreeBSD and OpenBSD have for the future. Currently they main their own Xorg trees, but I wonder if we'll ever see Wayland implemented on other platforms.