That sounds nice, but put into practice is, at best, marginally effective.
Democracy is actually quite good on a small scale without the element of compulsion, like your chess club or picking the most able-bodied man to lead the local volunteer fire co.
People are both more familiar with the people being voted for and can see a payoff for voting. There's more chance of dying in a car accident on the way to the polls in the US than of your individual vote mattering though.
Democracy such as in US politics, in comparison, creates a perverse incentive structure where it is more often the most suave liars rising to the top. The millions spent on Super PACs or stroking the media is chump change compared such things as awarding gov't contracts or the power of politicians and unelected bureaucrats to shape industry-changing regulation.
Can we vote out the worst ones? I guess, as long as we are talking about a more legitimate democracy than Russia or Venezuela. Still, even in the US people are going to elect pretty much the same thing every time they switch. There's very little difference between a Romney (or clones) and Obama.
What needs "voted out" is democracy as a whole. I realize that this cuts to the core of most Americans' religious worship of democracy, but hopefully people can at least consider the options—I suggest libertarian anarchy—and educate themselves about some serious criticisms of the system we have. I suspect I am going to get more downvotes than coherent objections, but oh well. See Hans Herman Hoppe's Democracy: The God that Failed or Bryan Caplan's The Myth of the Rational Voter