What worked for me, after my boss just put up a stock ad and got hundreds of replies then decided that they couldn't cope and handed the problem to me. First spam all the applicants and say terribly sorry there's been a transition in the team and now I am in charge. Enclose copy of new ad, in email emphasise that I have tier CV and want ~100 words explaining what they understand about the job:
- ask for a very specific thing in the cover letter. In our case, which version of the specific IDE they were familiar with.
- specify that we care about relevant skills, and having the legal right to work here. nothing else.
Of the 50-odd responses who survived a 10 seconds each filter (ie, did you answer the question), I skimmed the CVs and picked 10 I liked and 10 more I thought would be ok. I did this via a 5-bucket sort - as I read each email I dragged it into a numbered folder. Then I created folder 1a,1b and put 10 in each. Sure, that's about an hours work but it's easier than doing an online test and trying to make it non-gameable.
Interviews were a five minute chat then we dumped them in front of a computer with our development setup on it, and a series of programming tasks. Starting from "this button. Make it so when the user clicks it a dialog pops up saying 'click'" and going up to "there is a memory leak in this ~100 line command line program. Find it and fix it". They were asked to talk me though what they were doing, and while most problems followed each other from the same based, they started with a "perfect" solution to the previous ones at each step so that we didn't deviate too far.
I was pleasantly surprised at how effective the "brown M&Ms" question was, and how predictive the series of programming tasks was.
https://www.insider.com/van-halen-brown-m-ms-contract-2016-9