When you click and hold to move the map, and you scroll the mouse all the way off the map, the map acts like you are continuing to hold the mouse button down, even if you let go of the mouse button. When you return the mouse to the map, it acts like the button is still pressed and the map sticks to the pointer...for which you have to click again to get out.
This is unexpected behavior, especially if you are trying to scroll through more than one screen with repetitive mouse drags. The map scrolling should maintain the same behavior of the mouse button hold even if you scroll off the map.