Most of the delay is I/O bound. Handshaking happens over the Display Data Channel (DDC) which is a protocol over I²C (a two-wire comunication bus). HDMI/DP/DVI have 2 dedicated wires inside the cable for this, USB-C has to switch over to use differential signaling on pins A8 and B8, and then switch back after communication is done.
The bandwidth of that bus is small indeed, but most of the latency comes from artificial delays and error recovery in code: monitor is sending a message to host then sleeping 30ms to make sure the reply is ready, host doing the same thing, and this happens multiple times to negotiate pixel clock, signal timing, colors, resolution, additional features like suspend/reset/HDR/overscan etc.)
You can see how in the MCCS specification [1] both host and display entrypoints for communication start with a delay: https://shots.panaitiu.com/e8D7tQ
You can also see how that looks in Lunar's DDC code [2] [3].
For now, it's incredible that we actually have such a standard as DDC and most monitors work with most devices. A coordinated effort between monitor vendors and laptop/PC/GPU manufacturers into creating another faster standard with modern technology seems out of the question. But maybe with the advent of Thunderbolt this may change in the future.
[1] https://files.lunar.fyi/mccs.pdf
[2] https://github.com/alin23/Lunar/blob/master/Lunar/DDC/DDC.c#... : Delay on requesting data from monitor
[3] https://github.com/alin23/Lunar/blob/master/Lunar/DDC/DDC.c#... : Delay on communication error before retry