If I ever need to feel good about myself as a software developer, I only ever need to read this book's chapter about the X window system.
An office I worked in in the early 90's had a shelf with a ten(?) volume set of X windows books - and each individual book was a thick bugger. I'd only had fleeting experience of it, but remember thinking "how complex can this thing be ?!"
The architecture was layered, too, so you typically only ever used one book at a time. Doing raw graphics operations? All you need is the Xlib book. Xt/Xaw? There's a book for that. Motif? There's a book for that, too, but it wasn't in our set.
The raw X protocol was, IIRC, documented in volume 0.