Yeah, it sounds like you've got the right attitude about it. Personally, I failed to get tenure for a simple enough reason - I didn't publish enough and ended up leaving academia before going up for tenure (although after five years I could tell I was probably going to be short anyway). The reasons for that are pretty complicated and mostly tied to frustration at the mismatch between what I wanted to do (and thought professoring would be), and what it actually was.
Again - my teaching was a vanishingly small part of what actually counted at the Uni where I was a professor, and even my research work wasn't quite right. I wrote a lot of interesting software for gathering video data in classrooms (this was the mid 00s) and analyzing student-teacher interaction really closely to show that general models of learning and teacher activity were pretty thin in terms of how they align with actual classroom activity. None of the software or curriculum I wrote was valued in any way by my peers (didn't count toward a tenure case, only r1 journal pubs count).
I had a really hard time getting that work published (too techy for regular journals, to ed-dy for tech journals, and my absolute-favorite piece sat in review purgatory for literally 26 months before coming back with positive reviews and an editorial rejection for being not quite a fit with the journal - this is what ended up sinking me actually, if I had gotten that particular rejection two years earlier, I could have easily shopped the paper elsewhere and been in much better shape for a tenure case).
But ultimately, I was just not happy professoring, as it turned out to not be the job I wanted it to be. Teaching and research are interesting and fun, committee work and being pressured to write big grants and politicking for courses to be offered and all that were not what I wanted to do (stress and just time away from teaching, writing code and analyzing video). Plus side, I'm a lot happier now that I've figured all that out.