There are a couple of reasons I'm not a huge fan of plans like this. I have a friend running for state legislature under a similar goal (though only district residents would vote on his votes, not all state residents), and these are the two big points I made to him.
1. Most people aren't equipped to think about the kind of issues that come up with legislation. There are subtle interactions between laws, small economic provisions can have huge market effects, and many sentences end up meaning the exact opposite of what they seem to say. Ideally (for me, though you may disagree), the person getting voted in is equipped by experience, education, and general smarts. I want someone voting who is studying law, proposed and current, and making informed decisions. I don't always get that, but I'm more likely to get that with one person than with many.
2. Many Americans are terribly prejudiced. Pick any poll you have handy showing that more than 50% of Americans believe in something ridiculous and imagine them voting on a law related to that belief. Historically, perhaps, this vision is dimmer. It's a cultural issue, not an absolute one, so if we get better as a people, this argument against will become less important.
I do like when these kinds of ideas are brought up, however. Lots of interesting talk happens.