As Cletus wrote in his detailed comment, "this is the issue that will just never die." We discuss company hiring procedures here on Hacker News over and over and over, because most of us have applied for a job at least once in our life, and those of us who are building up a start-up have to think about whom we would hire.
The point in the submitted blog post by 37 Signals, "The only reliable gauge I’ve found for future programmer success is looking at real code they’ve written, talking through bigger picture issues, and, if all that is swell, trying them out for size," basically says that you should do a work-sample test to hire a programmer. And that's what research says. The more a hiring manager understands what a worker will do on the job, and the better the manager appreciates what a new hire may grow into doing after a few years on the job, the better the manager can devise a work-sample test to identify a good worker. There is a research base for this. Work-sample tests aren't perfect hiring procedures either--nothing is foolproof, and every hiring procedure goes wrong with both "false positives" and "false negatives"--but you can improve your odds of hiring a good worker by using a work-sample test routinely whenever you are hiring. As a job applicant, you can select better rather than worse bosses and co-workers by aiming most of your applications at companies that explicitly use work-sample tests as part of the hiring process.
With the help of several Hacker News participants, I have written a FAQ about company hiring procedures, revised over a few months of edits to fit the typical recurrent HN discussion of this issue. See
http://news.ycombinator.com/item?id=5227923
for the latest posting of that, with full references to the research literature and legal cases about hiring workers in today's world. Feel free to contact me through my HN profile
http://news.ycombinator.com/user?id=tokenadult
if you have suggestions for improving that FAQ before I post it to my personal website.
P.S. Even before I saw the prediction at the end of Cletus's comment, I planned to make the prediction untrue.
EDIT TO RESPOND TO FIRST REPLY ABOUT PUZZLE QUESTIONS: Yes, if a company in the United States insists on using puzzle questions as a hiring procedure, and justifies using those puzzle questions by saying that they want to see which applicants are "good problem-solvers," or "able to think on their feet," a rejected job applicant just might be able to subject the company to a very expensive lawsuit based on employment discrimination, unless the company has prepared beforehand a validation study showing that those puzzle questions have a bona-fide relationship to work requirements. I would not advise a company to take that risk, especially when the legally safer alternative of doing a straight-up work-sample test is available. The law is different in other countries, and as a reply in this thread points out, in the EU it is generally legal to use straight-up IQ-type tests in hiring processes, although those are underused by private companies in Europe, according to the sources in my FAQ post.
ANOTHER EDIT, TO LINK TO AN UNBELIEVABLE ANECDOTE ABOUT HIRING:
In August 2012, I heard a story from a hiring manager of programmers about the hiring procedure he uses as an initial screen for applicants who have degrees in computer science: "Write a loop that displays the numbers 1 to 100." That sounds awfully easy, even to me, but he says that the great majority of his applicants with accredited CS degrees fail that screening test. My earlier telling of the full anecdote
http://news.ycombinator.com/item?id=4603414
and his
http://news.ycombinator.com/item?id=4919749
seem pretty nearly unbelievable in what they imply about how clueless many CS grads are, and yet I think the anecdote is a true description of reality. Cletus too mentions in this thread, with considerable agreement from reply comments, that many people hired as programmers cannot actually program.