These results surprised me. I expected a much wider gap in correct move % between a 1500-player and grandmaster. It'd be interesting to see if the slope of the graph is steeper for minor blunders that reduce the evaluation by less than a pawn. These are the more subtle positional errors - weakening a square, not maximizing piece activity, wrecking your pawn structure, etc. Amateur games are filled with these mistakes, but they are much rarer in GM games, and I'd expect the difference to be more than just a few percentage points. But Crafty's not the right engine for this job. You'd want something with a more sophisticated evaluation function, like Stockfish (several-hundred ELO stronger than Crafty).
When Grandmasters Blunder: Even the best make mistakes
21–30 of 45 posts
Re: When Grandmasters Blunder: Even the best make mistakes
#22These results surprised me. I expected a much wider gap in correct move % between a 1500-player and grandmaster. It'd be interesting to see if the slope of the graph is steeper for minor blunders that reduce the evaluation by less than a pawn. These are the more subtle positional errors - weakening a square, not maximizing piece activity, wrecking your pawn structure, etc. Amateur games are filled with these mistakes…
Give us a few days :p. We'll have exactly the dataset you need to answer these questions. (And we'll be releasing it publicly.)
Re: When Grandmasters Blunder: Even the best make mistakes
#23Hi guys, author here. I'll be monitoring this thread for the rest of the evening. Happy to answer any questions.
What time controls were you looking at? I'm half-jokingly wondering if the big dip of correct moves in the upper 2800 range is Nakamura's crazy opening style :) (He's rated upper 2800's in blitz and rapid)
My guess: the only one that has been rated in that bucket in the last year was Caruana, post-Saint Louis. At that point, he suffered (and is still going through) a big decline with several bad blunders.
Re: When Grandmasters Blunder: Even the best make mistakes
#24Using "number of pawns of evaluation lost" as a proxy for the severity of the blunder has some fundamental problems. The main one is that the relationship between evaluation in "pawns" and expected result (expected value of the game result, from 0 to 1) is not linear. (It couldn't be, since one of them maxes out at one.) It's actually more of a sigmoid curve.
This means that a player may easily make a horrific "3-pawn blunder" reducing his evaluation from +8 to +5, but in fact all he's done is reduce his chance of winning from 99% to 98%. Actually, the +5 move may even be better in practice, in that it might lead to a sure safe win rather than a tricky blowout.
Even if you changed the definition of blunder from "reduces the evaluation by n pawns" to "reduces the expected result by x", I would have an issue in that it ignores any of the human aspects of blunders. If someone drops a pawn outright for no reason (eval change -1), that is a blunder because it was so trivial to avoid. But if someone, even a grandmaster, makes a move that causes a large drop in eval due to allowing a sacrifice that no human could calculate all the ramifications of, because as far as he (and probably his opponent) could humanly calculate it didn't lose, it is hard to call that a blunder. (Conversely, failing to see some immensely complicated non-forcing winning move may be unfortunate but it's not a blunder.) But that's more a cavil with terminology than a methodological error; the study is still measuring something interesting, just not quite what I think it is claiming to measure.
Re: When Grandmasters Blunder: Even the best make mistakes
#25Grandmasters blunder more often than this. I would venture to say that what correlates with blunders more so than rating is time. Error rate goes way up in Blitz and Rapid. IMO the more interesting thing about chess skill at the top is how much way way better GMs are than everyone else. To me, ratings at the top feel more like an exponential scale than a linear one. For example, I have beaten International Masters at…
Elo ratings are on an exponential scale, at all levels.
Re: When Grandmasters Blunder: Even the best make mistakes
#26These results surprised me. I expected a much wider gap in correct move % between a 1500-player and grandmaster. It'd be interesting to see if the slope of the graph is steeper for minor blunders that reduce the evaluation by less than a pawn. These are the more subtle positional errors - weakening a square, not maximizing piece activity, wrecking your pawn structure, etc. Amateur games are filled with these mistakes…
Give us a few days :p. We'll have exactly the dataset you need to answer these questions. (And we'll be releasing it publicly.)
Great! After thinking about it some more, I think I understand why the graphs are flatter than I expected. There are differing degrees of difficulty in tactical mistakes. When a 1500-player blunders a pawn or piece, it's often resolved by a trivial one-move sequence. GM blunders are more subtle, often requiring a lengthy (say 5-10 ply) sequence to resolve. You could prove this by recording the minimum search depth the engine needs to recognize the blunder. (This is tricky, because search extensions result in many sub-variations being analyzed much deeper than the nominal search depth, but I seem to recall that Crafty has an option for disabling extensions.)
Re: When Grandmasters Blunder: Even the best make mistakes
#27Grandmasters blunder more often than this. I would venture to say that what correlates with blunders more so than rating is time. Error rate goes way up in Blitz and Rapid. IMO the more interesting thing about chess skill at the top is how much way way better GMs are than everyone else. To me, ratings at the top feel more like an exponential scale than a linear one. For example, I have beaten International Masters at…
>To me, ratings at the top feel more like an exponential scale than a linear one. For example, I have beaten International Masters at chess lots of times but have never once beaten a GM. If true this is purely psychological. You are unable to beat a GM because he's a GM and you think you're unable to beat GMs. The strength difference between IMs and GMs simply isn't that great. Because the GM title is based on result…
I think you are citing the exception(s) to the rule. Most GMs are stronger than IMs imo. I don't think it's psychological. I have played players (GM and otherwise, including other untitled players like myself) that I know are so much better than me because they win and I can't even comprehend how they arrived at making the moves that they did.
As an aside, this is kind of an issue I have with chess analysis. A computer can 'verify' that a certain move is good or bad. That's fair enough. But in the past I have seen players (of lower skill level to me) discuss analysis in for example, a battle between two bigname players.
I have sometimes wondered if these discussions are truly honest because I have seen moves made by top players that I don't even understand how they arrived at the process of deciding that was the correct move vs others. Excluding GMs, a human simply cannot prune the game tree at depth like a computer can. So discussing a few tiny branches of the game tree like one is correct and the others aren't just seems really silly for the rest of us.
Re: When Grandmasters Blunder: Even the best make mistakes
#28> Due to cost limitations we had to limit crafty to 2 seconds of analysis time per move A grandmaster with standard time controls could defeat a 2-second limited Crafty. So how do you know you're finding true blunders, and not simply positions that the engine evaluates incorrectly?
This is definitely the biggest limitation of our approach right now and there are certainly some things that we counted as blunders that aren't true blunders. We're working on rectifying this by doing another pass with a better engine and more time to analyze. That said we tested this on a smaller set of games by comparing it to results from better engines and found that only a very small number of moves tricked craf…
you could just rewrite your article to call these "obvious blunders" - i.e. which you define as ones that crafty identifies in 2 seconds or less. redefine what you're doing so your methodology is correct :) Plus it's still interesting. Probably more interesting than blunders that take longer to identify!
Once you have found the blunders, you can verify them by analyzing the found positions more deeply. (Of course you should also report the number of false positives - ones that appear blunders after 2 seconds but turn out not to be on slightly longer analysis.)
Re: When Grandmasters Blunder: Even the best make mistakes
#29It seems to me that the article overlooks one glaringly obvious issue: that the two blunders may not be independent events. In this case, it seems quite likely that the second player's blunder was made much more likely by the fact that the first player had just blundered. To be more specific, white moved the king which appeared (at first glance) to prevent black from using a check threat to attack white's rook. The b…
This is a valid point. We're going to be publishing some data in a few days that will have everything you need to test this hypothesis.
Re: When Grandmasters Blunder: Even the best make mistakes
#30Using "number of pawns of evaluation lost" as a proxy for the severity of the blunder has some fundamental problems. The main one is that the relationship between evaluation in "pawns" and expected result (expected value of the game result, from 0 to 1) is not linear. (It couldn't be, since one of them maxes out at one.) It's actually more of a sigmoid curve. This means that a player may easily make a horrific "3-paw…
Yup, this is another very valid criticism. I think the answer is probably to have a cutoff on the lower bound. Basically saying that for a move to be a blunder it has to leave below a certain absolute value, maybe +2 pawns, in addition to a certain amount below the best possible move.