I actually already wrote this as a reply to another comment, but I think it merits a comment of its own. The best way to do the scoring for the applications is to actually make it into somewhat of a game. Each application should be given a starting elo of 1200. Every time a user wants to go and review applications they will be presented with two separate applications with a similar elo and then will be asked to vote…
Would this be implemented as only showing pairs within a certain threshold of difference, or would it choose an application purely at random and then the application with the closest score (or even 50/50 the next highest or next lowest)? The former seems like it could lead to cases where applications end up in a void between clusters and stop appearing in votes, preventing late voters from giving an opinion on them.…
Each potential implementation has slightly better/worse corner cases, but the general quality of the ranking should still be extremely accurate regardless of the particular implementation. It takes surprisingly few comparisons for items to find their appropriate ranks.