Human activity weathers stone monuments faster than desert winds. If I recall correctly, the original sheathing stones and capstones of the pyramids were gradually stolen. The Sphinx was used for target practice.
Water, on the other hand, will tear through your stones much more quickly. In a wet, temperate climate, the weak acid falling from the sky will wear limestone. The moisture invading cracks and freezing will widen those cracks. Lichens will bore tiny holes into the rock, and plant roots will cause fissures to expand. Additionally, moisture that penetrates to metallic structural reinforcement will oxidize and swell that reinforcement to pop bits off of the stone (observable from earlier attempts at restoration at the Parthenon, where the replacement anchor pins weren't coated in lead correctly, and they started to crack the rock blocks--now they use non-corroding titanium pins). Sometimes mortar does not swell or expand at the same rate as whatever it binds, and something will crack, admit water, and generally deteriorate from there.
Water is the enemy of preservation. They pyramids didn't survive just because they are stone, but because they are also dry.
And on a long enough geological timescale, every potential building site will eventually experience every climate, including being entirely immersed in salt water.