It's not a talent shortage, it's a hiring problem

The other problem that I found annoying when interviewing (even with Startups) is the way the whole system is organized.

1. Talk to recruiter who is a non-tech person who doesn't really know what they are talking about and to whom you cannot really go in deep. This is the only casual conversation that you get to have about the company. (Assuming you don't know anyone there..)

2. Have phone screens which are that exactly: Ways for the company to give you a few puzzles and determine whether you based on whether you follow a certain process of solving them are actually good enough to join them. At the end of this, you are asked if you have any questions for them. In a big company, this person might not even ever work with the team that you are interviewing for. There is a good chance that after a grilling interview, you are tired and can't really have an actual conversation.

3. You have interviews at their office which again are designed to grill the candidate and don't offer any time for the candidate to get to know the people on a personal level.

I used to wish there was a way I could talk to an engineer in a company over coffee (and I am not talking about the whole lunch interview thing that a lot of companies do) and figure out what they actually do, what they want in a candidate and what sort of skill sets that they are looking for. Over the past few months, I have come to the realization that I am not sure many companies actually know the answer to that and all they are trying to do is to get the "best" without actually properly defining what that even means.

"There's a misconception that anyone can hire. Tech companies are outsourcing their IT to heroku and AWS because programmers can't sys-admin. They hire lawyers and accountants to take care of legal and financial matter. But hiring? "I can do that" they think."

In their semi-defense, they probably can do it at least as well as the vast majority of supposedly specialized technical recruitment firms, who are also largely a joke (yes, there are a few exceptions that prove the rule).

The problem has been solved long ago by other industries, see lawyer and doctors. Eg: entrance qualification + no H-1B competition for jobs.

I agree completely. This is especially true when the KEYWORD-HEAVY-SPECIFIC-TECH is something that isn't easy for a programmer to gain meaningful experience in outside of work, like cloud computing, distributed software, or MapReduce architectures.

I like the point about how randomness seems to be such a big part of the process. If you're not using connections and are applying through normal channels, to me it seems like there is so much that can go wrong that isn't related to your ability to do the job. Maybe the recruiter had a bunch of email that day and missed the application. Maybe the person who screened your email didn't like the font or design of your resume.

To me, one of the big problems in the process is the lack of a feedback loop. Since recruiters and interviewers rarely give honest feedback (probably for good reason), it is hard for an applicant to know if they're applying to the right jobs or not. And faced with this lack of information that would otherwise allow you to be more effective and only apply to a few jobs that would be a good fit, the strategy becomes just spamming your resume everywhere until you get a response. On the company's end, this means that you have this extra mass of applications of wildly varying quality because no one has a clue what you're really looking for.

How could this feedback loop be improved? I'm not sure.