This is a tough decision, but you're absolutely not alone.
I've dealt with this in founding a company with my closest friends. I didn't agree with many aspects of how the company was being run and simply picked up and left. I also owned a significant portion of the company in my departure.
In my opinion, you keep the shares that have vested to you, not the shares you are entitled to. I left 2.5 years after signing my stock purchase agreement, so a large percentage of shares were already allocated to me. They tried to get me to give back some of my shares, but I worked extremely hard and sacrificed a lot over the 2.5 years (including working for free), so the shares were the only feasible form of compensation I could argue for.
Eventually what happened was the company issued more shares, effectively diluting my holdings to nothing.
They're still puttering along, doing the whole "I'm the CEO, bitch" thing, we're all still close friends, and I honestly don't regret a single thing about my decision because I was miserable working on it and fighting with my best friends every day.
Do what you feel is right and what will make you happy. In the end, that's what's important. Keep your shares. The chances of them being worth a significant amount are low, but in the event they're worth anything, you've earned that payout.