I don't work in academia, but at my current workplace, we've removed the "Neutral" rating in our interview feedback system. You're required to pick a side, and you need to back up your position. If you really can't form a recommendation, the candidate has to do another interview (I haven't personally seen this happen). That said, we also are able to set a maximum # of interviews per week that we will accept, and our…
If you know anyone in the crowd, it's a good idea to 'seed' the crowd with prepared questions or ask friends to kick off the q&a portion with a question of their own. We do this when we have guest speakers and it greatly improves the moral of the speakers and everyone goes home happier.