Note: what follows is probably not how anyone actually does it. It is just an illustration that adaptive video is not incompatible with E2E encryption.
Suppose you have a block of 4 pixels, represented by 4 24-bit values. Instead of sending the 4 pixel values, send one 24-bit value that is the average of all 4 pixel values, and then the actual 24-bit values for 3 of the 4 pixels. The receiver can figure out the 4th pixel from those 3 and the average.
Send the average values and the groups of 3 discrete pixel values in logically separate streams, separately encrypted.
If something transporting this needs to lower the bandwidth, it can just drop the E2E discrete pixel stream, leaving just the E2E average stream. The receiver can then use that average value for all 4 pixels, in effect getting a video that is 1/2 the resolution both horizontally and vertically.
This scheme only gives you two rates: Full resolution and 1/2 x 1/2. No doubt you could do systems based on block sizes other than 2x2, and with multiple levels of averaging, that would give a wider range of fall backs.
Actual state of the art video encoding is, I believe, based on things like the discrete cosine transform, which represents an image as a sum of cosines of various various frequencies.
In this kind of representation the higher frequencies correspond to higher resolution detail in the image. I'd expect that you could do an E2E transmission scheme were you have different encrypted streams for different frequency ranges. Like with my far less sophisticated or clever 2x2 averaging scheme above, you could simply drop the streams for higher frequencies and the receiver would be able to reconstruct a lower resolution image, but unlike my 2x2 averaging scheme this would have much finer drops in resolution.