> To be useful for discriminating between competent candidates,
I helped redesign the code challenge at one of the companies I worked for. We (the people across the different technical disciplines we hired for) put a lot of effort into ensuring that:
* the challenge could be completed in 4 hours
* the challenge resembled the sort of work people would do on the job
* the goal of the challenge was to inform a follow up interview, not punish people
* candidates were explicitly told that they were not being timed, but that we expected the challenge should take them up to 4 hours to give an idea of the upper-bound level of effort expected
The single biggest challenge was balancing "what do we need to know" versus "what would we like to know" versus the amount of time we were asking of our candidates.
We didn't hard-fail anyone unless they were blatantly not a good fit from their submission, and I habitually wrote multiple pages of feedback for them so they at least got something for their time if they didn't get a follow up interview.
Again, the purpose of the challenge was not a screen, so much as it was to help guide the in-person interview and so we had something concrete that we could discuss.
Follow up edit:
Designing code challenges is hard. You have to avoid something so objective it can be copy-pasted from stack overflow, but objective enough that it can be graded free from personal bias (as much as possible anyway). On top of that, you can't test for MVC+CQRS+FP+SOLID+every-other-possible-thing under the sun, because you have to be respectful of candidate's times.
The previous code challenge (the one I replaced) would frequently take people 8-20 hours to complete, and it was actually pretty simple. As you rightly pointed out though, candidates often read far more into the requirements than were actually there, and would often over-achieve in an effort to stand out. Not only would those submissions waste their time, it wasted ours as well as they took longer to develop feedback from.