Or perhaps you shouldn't get defensive of critique on your "design vision" such that you just write them off entirely like it has no value? You might like driving in the opposite lane, but rules say otherwise, and best practice can be likened to it because it's been decided on or landed that way due to multiple people being involved in the process of deciding those rules. Just as with best practices. There are best practices in design too, you know.
Anyhow. Depending on the viewport your panels do get in the way, it's 1/3 of the screen on mine for example. If you're going to hide the panels, hide them properly. That you show them at the top and bottom automatically, fine, though they could be smaller on mobile. However, you could at least consider the UX of using it on smaller viewports. If I want to scroll up a little bit while reading, they pop into view, making me have to scroll down to hide them again. Meaning I have to scroll higher up than intended, so I have enough space to scroll down. Another common user pattern is to slowly continually scroll too, and a tiny jitter upwards causes them to reappear, but requires comparatively massive downwards scrolling to hide again. Your site is not the only one to do this, but it's just bad design. Vision or no, and it's no justification, just an excuse. And then, what about left handed people? It's easier to accidentally click into the message box field, which opens up the keyboard and you'll have to close it. Pair this with it taking up the majority of thumb space for scrolling, given its size, and, well. Is it unreasonable for people to tell you it's aggravating? No. And this is without considering, and doing the minimum for, people who may suffer with controlling devices for whatever reason. Not by "bending over backwards" as some would have it, but at least taking such into consideration.
It's fine to have a vision of your intended or wanted design, but it's also important to be willing to reflect on how your vision may not necessarily be good design, depending on how one judges it, rather than just deflecting critique.