A Kaggle Grandmaster cheated in $25k AI contest with hidden code • Hacker News Board

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#151

Post by C1sc0cat » Thu, Jan 23, 2020, 12:36 PM UTC

starpilot wrote: post #41

sandGorgon wrote: post #6

I remember reading about this on Twitter because of a reply from h2o.ai account. https://twitter.com/ppleskov/status/1215983188876709888?s=19 The person was originally employed at h2o.ai and as a consequence of this was fired. Not sure if that was completely appropriate. Wasn't this a personal participation? Or are there "company teams" on Kaggle ?

At will employment.

"bringing the company into disrepute" is gross misconduct and normally a firing offence, even in countries with more liberal employment laws than the USA

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#152

Post by harry8 » Thu, Jan 23, 2020, 12:48 PM UTC

Hercuros wrote: post #128

harry8 wrote: post #107

Earlier quoted context omitted.

There is no such thing as "training code". You have a model that you have trained on some provided data - the training set. You give kaggle this model. Kaggle grades your model on some different data your model has never seen. The better your model classifies this data it hasn't seen the higher it scores and the more money you win. So again if you trained your model (code) on a training set that you have illegally ob…

People are saying that you do not just submit your trained model to Kaggle. You also submit the code that was used to train the model from the training set, which is used in the winning models to train them from scratch on the training set. Of course, that wouldn't have prevented this type of cheating of course, but it does mean that you can't submit a model that was trained on your own private data set.

You can overfit a model to your hold out set quite easily with repeated trials. It's a trap you have to avoid in normal circumstances! (Feynman: "You are the easiest person to fool"). Even if you have to submit code to generate your model parameters from "the training set" (which hasn't been explained at all well by "People" if that is indeed the case) you could do that overfitting deliberately here with the illegal unseen data as your hold out set. Aside from the advantage of a bigger training set. Aside from the advantage in model selection, which is not done with code from a training set. Aside from the advantage to your feature engineering also not done in code. Aside from the advantage to your regularization choices, bias parameters etc etc.

So yes you absolutely /can/ submit a model trained on your own private data set even if what you submit is a model code that will be re-trained. Even if "the training set" is different to the provided - you still have that scraped data so you can slice it up with the provided training set so that any selected training set does well against the rest. Now the overfit you've just carefully engineered should win against the honest models unless you suck, right? It's kind of risible that they had to go further and hard code certain results, don't you think? Perhaps if they still couldn't win with scraped, illegal additional data then everyone else had illegal data too? Perhaps Kaggle is not a good indicator of how good ML techniques are in practise? Perhaps Kaggle systematically overstates ML effectiveness due to this kind of uncaught cheating in many of their competitions? I bet kaggle won't look too hard at that.

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#153

Post by luma » Thu, Jan 23, 2020, 12:50 PM UTC

ribs wrote: post #87

It really worries me how many people are so quick to forgive him and tell him so. In my family if someone cheated they got called a cheater and suffered consequences. At least, they would have, if someone did something like that. But my parents didn’t raise mendacious villains. Look at this crap on Twitter: “Everyone makes mistakes. Thank you for the apology”. “Kagglers will still love to have you back” “It's great t…

I'm confused by this reaction here - this was a _brilliant_ hack of the system and I think his work should be celebrated. Was it in keeping with the intention of the competition? Of course not. Were lives threatened by their creative solution to the competition? Also no. At the end of the day it was a fun, inventive approach to a made-up problem.

So what's the problem?

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#154

Post by RandomWorker » Thu, Jan 23, 2020, 12:54 PM UTC

7777fps wrote: post #146

paulgb wrote: post #17

For an HN audience, the "How Bestspotting cheated" post on Kaggle may be a better place to start: https://www.kaggle.com/bminixhofer/how-bestpetting-cheated

It's an odd way to cheat too, if they realised they had data from the validation set, couldn't they have over-trained a model with the validation set in the training data?

Agreed, this actually happened in the buildings ASREA comp. the data was found online.

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#155

Post by bleuarff » Thu, Jan 23, 2020, 12:57 PM UTC

luma wrote: post #153

ribs wrote: post #87

It really worries me how many people are so quick to forgive him and tell him so. In my family if someone cheated they got called a cheater and suffered consequences. At least, they would have, if someone did something like that. But my parents didn’t raise mendacious villains. Look at this crap on Twitter: “Everyone makes mistakes. Thank you for the apology”. “Kagglers will still love to have you back” “It's great t…

I'm confused by this reaction here - this was a _brilliant_ hack of the system and I think his work should be celebrated. Was it in keeping with the intention of the competition? Of course not. Were lives threatened by their creative solution to the competition? Also no. At the end of the day it was a fun, inventive approach to a made-up problem. So what's the problem?

That's not a hack, that's blatant cheating, their solution litterally looked at the anwsers. It bypassed the ML model prediction, so that's not a ML solution, which to my understanding was the constraint of the competition. And in the end, that solution is useless for the adoption site, since the objective is to get adoption predictions (the animal has not been adopted yet). I'm confused too, by how can anyone think it's an acceptable behavior.

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#156

Post by luma » Thu, Jan 23, 2020, 1:02 PM UTC

bleuarff wrote: post #155

luma wrote: post #153

Earlier quoted context omitted.

I'm confused by this reaction here - this was a _brilliant_ hack of the system and I think his work should be celebrated. Was it in keeping with the intention of the competition? Of course not. Were lives threatened by their creative solution to the competition? Also no. At the end of the day it was a fun, inventive approach to a made-up problem. So what's the problem?

That's not a hack, that's blatant cheating, their solution litterally looked at the anwsers. It bypassed the ML model prediction, so that's not a ML solution, which to my understanding was the constraint of the competition. And in the end, that solution is useless for the adoption site, since the objective is to get adoption predictions (the animal has not been adopted yet). I'm confused too, by how can anyone think…

Their solution made use of the data available to them in a "prize" that was nothing more than a made-up competition for fun. Did the rules expressly say somewhere that one could not make use of available datasets?

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#157

Post by Townley » Thu, Jan 23, 2020, 1:04 PM UTC

luma wrote: post #153

ribs wrote: post #87

It really worries me how many people are so quick to forgive him and tell him so. In my family if someone cheated they got called a cheater and suffered consequences. At least, they would have, if someone did something like that. But my parents didn’t raise mendacious villains. Look at this crap on Twitter: “Everyone makes mistakes. Thank you for the apology”. “Kagglers will still love to have you back” “It's great t…

I'm confused by this reaction here - this was a _brilliant_ hack of the system and I think his work should be celebrated. Was it in keeping with the intention of the competition? Of course not. Were lives threatened by their creative solution to the competition? Also no. At the end of the day it was a fun, inventive approach to a made-up problem. So what's the problem?

For this particular problem, they were working on pet adoption timing prediction algorithms. The proposed solution wouldn’t work as efficiently in production as one of the other competitors. That leads to inaccurate predictions on pet adoption times, which increases costs for adoption centers, and maybe euthanasia rates among kill shelters. So animal lives would be impacted by this cheat.

More importantly, Kagggle does competitions across dozens of industries. If a culture of getting better at hiding your cheat pervades the platform, that could impact finance, transportation, and medical research. In those scenarios, lives would either be threatened or at least subject to sub-optimal systems.

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#158

Post by dlkf » Thu, Jan 23, 2020, 1:13 PM UTC

A large number of commenters fundamentally misunderstand what happened. They are saying "why did he upload his scraped training data? Why didn't he just train on it and upload the resulting model?" If you are making this argument, it means that you don't understand the contest. User rahimnathwani explains:

> In this competition, the training code was run on Kaggle's system, so you'd still need to smuggle in the extra data.

The question then becomes, how do you smuggle in the data? This is a much more interesting discussion than pontificating about the ethics of Pleskov's actions. In particular, a better understanding of this problem could have ramifications for how Kaggle could combat hacks of this variety. (By contrast, "shame on him" and "aww but he's a nice guy" are both useless, except perhaps as a form of virtue signalling).

It's essentially a cryptography problem. Does anyone know if this has been widely studied?

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#159

Post by nomadluap » Thu, Jan 23, 2020, 1:14 PM UTC

luma wrote: post #153

ribs wrote: post #87

It really worries me how many people are so quick to forgive him and tell him so. In my family if someone cheated they got called a cheater and suffered consequences. At least, they would have, if someone did something like that. But my parents didn’t raise mendacious villains. Look at this crap on Twitter: “Everyone makes mistakes. Thank you for the apology”. “Kagglers will still love to have you back” “It's great t…

I'm confused by this reaction here - this was a _brilliant_ hack of the system and I think his work should be celebrated. Was it in keeping with the intention of the competition? Of course not. Were lives threatened by their creative solution to the competition? Also no. At the end of the day it was a fun, inventive approach to a made-up problem. So what's the problem?

The problem was that there was a prize at stake, and everyone else was playing by the rules.

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#160

Post by gpm » Thu, Jan 23, 2020, 1:24 PM UTC

sundvor wrote: post #111

ribs wrote: post #87

It really worries me how many people are so quick to forgive him and tell him so. In my family if someone cheated they got called a cheater and suffered consequences. At least, they would have, if someone did something like that. But my parents didn’t raise mendacious villains. Look at this crap on Twitter: “Everyone makes mistakes. Thank you for the apology”. “Kagglers will still love to have you back” “It's great t…

I'm thinking Kaggle should look to Epic/Fortnite for how to handle cheaters. A lifetime ban would not be out of the place here, considering it went on to win the competition with no admission until caught.

Kaggle should be looking to the police on how to handle cheaters. He stole 10,000 dollars, he defrauded h20.ai, etc. That's "go to jail" level crimes, not just "get banned from a competition and fired" level crimes.