Earlier quoted context omitted.
I didn't see any classification measures besides accuracy... Could You provide the confusion matrix? It's easy to derive all the important values (FPR, TPR, precision so on) and get a lot better idea of quality. Since the outcome is either success or not, the coin flip classifier yields an accuracy of 50%. It would be worth noting that in the paper.
It's true that that should have mentionned it in the paper. The basline here is always telling it fails, which gives you about 52% accuracy. I should definitely provide the confusion matrix, it's on my TODO list. I will add a performance section to the website where I show this kind of information.
I didn't mean to put down the article. Quite the opposite. It's truly an amazing application and a well thought through execution of the machine learning methods.
I have been wondering for some time, what the supervised learning could be useful for, which could benefit the broader audience (beyond computer vision, gaming, medicine and all the much popular and repeated applications). That You nailed perfectly.