Earlier quoted context omitted.
I'm essentially looking for something like imgui but with ability to draw directly to framebuffer. I looked at egui but that has same issue.
Imgui doesn’t write to a framebuffer, it records GPU commands. Really embedded MCUs don’t have space for a framebuffer either, a technique used is filling a line buffer on a timer interrupt and driving the lcd off that. The gui has to be written quite differently.
Beefier display modules will have a framebuffer inside so you don’t have to mind the timing, even if you still have to transfer the image line by line because a full screen buffer (at the full colour depth) may not fit into the microcontroller’s own RAM. It would actually be interesting to do some sort of IMGUI for this kind of environment, that would perhaps rerun the display function multiple times per screen update, once for each line where a widget’s bounding box starts or ends.