I remember this as well (I still have mine in fact!). However, I didn’t really care that much considering how much better the Game Boy Advance SP was than all the previous Game Boys. It was the first Game Boy to have a backlit screen, which meant I could finally easily sneak and play on it in the night without alerting my parents (instead of trying to do the same with the much less effective wormlight accessory I had…
I still have my GBA SP, too, but I'm pretty sure part of why charging it is so finicky now is that I used the headphone adapter so often. Which is a bummer. But it still charges with a little bit of a wiggle and I've got a flash cart with a bunch of translated versions of Japanese games. It's still a bunch of fun and I bring it when I fly.
Also, if you're into music and have a GB flashcart, lsdj is a must buy.