I think we can learn from some other (not-so-obviously-related) disciplines.
The game of go[0] has a ranking system where players progress from 30kyu (complete beginner) to 1kyu, then 1dan to 9dan (very strong).
[0]: http://en.wikipedia.org/wiki/Go_ranks_and_ratings
It's mostly statistical and based around comparing your skill level to other players. While a 7kyu level is not necessarily that well defined, and might differ between different ranking systems, there's still a meaning to the difference between the levels. When a 5kyu (stronger) is playing a 7kyu, the 7kyu player is given 2 stones in advance (handicap) to make the game even. The number of stones given as a handicap is determined by (or defines) the difference in levels.
It's not immediately obvious how such a system could be adapted for programming skills. But the idea of having levels, and having ways to measure one's level, is very interesting, and might be quite useful.
Japanese (and probably other languages) have standardized tests[1] to determine one's language abilities.
[1]: http://en.wikipedia.org/wiki/Japanese-Language_Proficiency_T...
You don't take a test to determine your level. Rather, you take a test designed for a particular level. You either pass or fail.
N5 is the lowest level (beginner)
N1 is the strongest level (near-native or even better maybe)
When you feel you are at say, level N3, you take the N3 test, and if you pass, then you can say that you have passed the N3 test. If you fail, you just fail. It doesn't mean you are N4 or N5. (As far as I know - I might actually be wrong here).
I wonder if some standardized tests can be, in principle, constructed to measure programming abilities.
The tests must be crafted such that, even if all the questions are known to the public, then studying for the test is the same thing as studying to build up your skill level.