Matasano used to bet the farm on it, I did it at my last job, and Latacora's doing it again. I don't think what those companies do is trivial. (Disclaimer: I'm a Principal at Latacora.)
In my experience, when you think you can't measure a thing in a work-sample setting, it means you haven't analyzed it. And, if you have no idea what that thing means, how could you possibly hope to interview for it?
For example: "can this person build a team and develop a solution to a complex technical solution" (that's several things, but OK, let's take a stab at it):
- Someone who can build a solution in a specific domain has to be able to analyze the problem into bite-size pieces. Given a complex problem description, write out tickets that need to get done. Can they identify sensible milestones, objectives and key results? Are the tickets roughly equally sized? Are the tickets self-contained? Do they come with an objective measure of "done"?
- Someone who can build a team to deliver a solution has to be able to estimate appropriate resourcing for a project. Given a complex problem description and a current set of resources, come up with a plan forward. Measurements include: a) Did they identify missing roles? b) Did they write clear job reqs with evaluation and hiring plans? A gazillion hiring managers couldn't write a clear job req to save their lives. c) Did they consider which positions may be effectively contracted out? d) Can they estimate what the P&L impact of this hiring plan is? Knowing how to read P&Ls and cost of development is a skill. e) Can they identify within their own plan which roles are really critical and which ones are nice-to-have? Being able to negotiate in the face of limited resources is a skill.
- Someone who can lead a team to deliver a solution to a complex problem has to be able to analyze when things go awry on a low-level. Put them in front of a PR where someone has subtly misunderstood a poorly-written ticket. Give them the context they need to understand why the PR is wrong. See how they tell people the PR is wrong, and how they react when people persist?
- Put them in a position where they have to talk to an employee who isn't doing well. Can they figure out how to be empathic while remaining professional? Did they say something that will get you sued? Opposite situation: put them in a room with a rockstar developer who's been harassing or badgering one of your employees. Can they speak to that person professional? Can they write an HR file note with a follow-up plan?