A lot of Chesterton’s Fence going on here… I remember thinking this way when I still had a pretty low-level understanding of music (like, “I’m reading arbitrary pitches and pushing the matching keys on the piano”), but once I started understanding bigger musical concepts/chunks, the reason for the notation became clearer, and now I see why it’s a pretty good local maximum for readability.
The majority of music uses a system of intervals called the major scale, which is seven notes with a particular spacing that is determined largely by psychoacoustics (that is, we didn’t just make it up). That spacing can be shifted around to any point on the underlying 12-note system, which gives different keys/modes. On any single piece of music you’re typically using only one of those, so those seven notes are the “skeleton” underlying it all. This is what it means to say “this piece is in D major”. (BTW, a fifth is the fifth note in the major scale, so yes, there is a 5!)
Standard notation privileges these modes of the major scale, which means (assuming you know how to play all the major scales on your instrument) there’s a 1:1 mapping from places on the staff to scale degrees. That means most of the time you’re reading straight off the score into the scale your fingers have memorized.
The key signature is what makes this work. You can even instantly transpose by ignoring the written key signature and reading in another key. Chords are also made readable because they’re usually based on thirds, which happens to be the interval that makes the notes stack up exactly next to each other, so a triad, seventh chord, etc. is instantly recognizable.
So basically the notation is optimized for the common case (the music is using notes from a mode of the major scale) and makes exceptions (accidentals) stand out.
And yes, pianos are a pain because all the (musical) keys are different on the (physical) keys. However, pianos are great because all the octaves look exactly alike (so finding all the Ds on the keyboard is trivial). Stringed instruments like guitars or violins are the opposite: all the (musical) key fingerings are the same, but you have to memorize the positions of all the notes because they all look the same. Pick your poison.