Why less competent may rate their own ability higher than more competent

A common "why" theory that I personally subscribe to: the less competent don't know enough to know what they don't know.

Those who want to be truly great should be humble. They should also check themselves as much as possible with objective measures and feedback from trusted and respected fellow practitioners.

Come to think of it, I suspect this phenomenon affects the implementation of programming languages. A lot of second tier CS talents implement languages because they have enough ego to get through the ordeal.

"Their misconception is fueled by the fact that they are completely unaware of their own ignorance. This unawareness or unwillingness to admit their ignorance subconsciously prods them to make (wrong) assumptions in order to fill the void in their knowledge."

"Ignorance more frequently begets confidence than does knowledge."

"What we do not understand we do not possess."

"We're even wrong about which mistakes we're making."

I once worked with a guy (not in an engineering discipline) who could talk circles around much more competent people. He knew all the buzz words, and even knew most of the topics about one level deep. When he spoke, he spoke with an absolute air of authority on whatever subject he was talking about. He managed to convince division manager after division manager (he jumped around a lot and it was a very large company) that past failures were just because of circumstances out of his control.

However, he didn't actually know anything and every project he was put in charge of either failed disastrously, or was caught from the brink at the last minute by colleagues who worked double shifts to do the project (which of course allowed him to point to those same colleagues as incompetents that he had to deal with on his project and hence the reason for failure).

The really sad thing is, I could never figure out if he knew he was grossly incompetent and just spinning things to keep a job far above the level he should have had, or actually believed the yarns he wove about what he knew.

I would have guessed that if someone impresses HR, you could find out if they really know their stuff by getting a KNOWN competent person to ask them lots of hard technical questions, make them solve problems on the spot, etc.

A friend recently gave me a great example of how to do open-ended questions and probe someone's knowledge: ask a question like "when you type in a web address and hit enter, what happens?" The answer can range from "your computer loads the page" to delving into DNS, IP routing, HTTP headers, server load management, server-side processing, sessions, database access, cookies, browser rendering, etc etc. If the candidate gives a simple answer, you can continue to probe different areas to see how much they know in each one.

The trick is that the interviewer has to be competent enough to tell what the candidate knows and whether he/she will BS rather than admit not knowing something (which is also important to find out).

If you don't already have anyone competent enough to conduct the interview - maybe you're hiring your first programmer - you could temporarily hire someone who comes highly recommended to help you with the interview process.

So, was this guy vetted like this on technical merits, or did he just talk impressively to non-technical people? I'm curious to know how he slipped through.