Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Kaggle Grandmaster cheated in $25k AI contest with hidden code

theregister.co.uk

Post reply on HN
Print view View original

101–110 of 197 posts

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#101

Post by niceworkbuddy »

My two nuggets. This reminded me of Ijon Tichy's saying after one of his voyages:

" Thus concluded one of the most unusual of my adventures and voyages. Notwithstanding all the hardship and pain it had occasioned me, I was glad of the outcome, since it restored my faith, shaken by corrupt cosmic officeholders, in the natural decency of electronic brains. Yes, it’s comforting to know, when you think about it, that only man can be a bastard. "

(source: The Star Diaries, Stanisław Lem)

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#103

Post by joe_the_user »

sytelus wrote: post #75
mannykannot wrote: post #65

Earlier quoted context omitted.

"Winning" by anything that can reasonably called cheating, as in this case, does not advance the general state of the art. Innovation is best served through appropriate rules and competition structure.

Yes, and that's the right thing to do in the academic research setting ("advance the state of the art"). But the public competitions with monetary rewards are not the same setting. I can imagine scenarios where the guy stole the test set from Kaggle servers (i.e. unlawful access) should disqualify him permanently. But the essence of the competition should be the focus on cracking a given problem, not about a specific…

One test a good of ML competition: Can it be solved by simply hiring lots of humans to make predictions without incurring significantly more costs than the prize money?

What value to the organizers, to society or to whatever are you imagining coming out of a free-for-all style competition?

I think the organizers now imagine that the result would identifying good, generic prediction algorithms along with identifying good AI programmers capable of producing general prediction algorithms.

It seems like the contest framework already has become a bit problematic through context winners just being good at contests and not otherwise achieving anything.

But what are you thinking of? There are already hacking competitions btw.

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#104

Post by alanfranz »

vladislav wrote: post #73

They would have able to win and get away with it if they incorporated the knowledge of the external dataset directly into the ML model, provided they had a reasonable estimate on the fraction of overlap between the external data and the test set. A weak version of this would be to just train on the external data in addition to the provided data. A stronger version would train regularly on the provided training data a…

This.

You've got the testing set. Create random HPs and tune them to fit. The way they cheated is stupid.

And the way the testing set can be obtained is silly.

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#106

Post by raverbashing »

ribs wrote: post #89
ribs wrote: post #87

It really worries me how many people are so quick to forgive him and tell him so. In my family if someone cheated they got called a cheater and suffered consequences. At least, they would have, if someone did something like that. But my parents didn’t raise mendacious villains. Look at this crap on Twitter: “Everyone makes mistakes. Thank you for the apology”. “Kagglers will still love to have you back” “It's great t…

I’m not saying that Pleskov is a sociopath, but I’m beginning to see how sociopaths manage to make their way in the world without getting punished.

You're not too far off, it usually starts with the family attitude.

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#107

Post by harry8 »

iudqnolq wrote: post #95
harry8 wrote: post #62

Earlier quoted context omitted.

They scraped data from the website of the org that wanted the results and funded the competition. That data was supposed to be unseen for the competitors and used to grade the models. This cheat was to use that scraped data in its training set and, beyond that, hard code some predictions. It's looking up the answers in the grading sheet while taking an exam.

Your parent is saying kaggle takes the training code and runs it themselves. This makss secretly training on the known right answers impossible.

There is no such thing as "training code".

You have a model that you have trained on some provided data - the training set. You give kaggle this model. Kaggle grades your model on some different data your model has never seen. The better your model classifies this data it hasn't seen the higher it scores and the more money you win.

So again if you trained your model (code) on a training set that you have illegally obtained. That is broken the rules of the competition to get the additional data. Data that is also going to be used for competition verification and grading, then you have cheated. Doubly so if you just hardcode outputs to boost your score, which they did here.

They took the official training set. Said, we need more. And scraped websites to get a bigger, illegal training set. This is against the rules and is cheating. They got caught.

It really is equivalent to looking up the answer sheet while taking an exam.

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#108

Post by raverbashing »

bitxbit wrote: post #3

Never understood Kaggle. The Netflix Prize was great. Now it’s just people gaming Kaggle to get a job.

I don't see a problem with that. It seems to me most companies sponsoring the contests have legitimate problems they wanted solved.

Now it really illustrated to me how data science is still an incipient field. Even the basic Titanic example has a lot of fake solution and a bit of "cheating" (overfitting).

And don't get me started on the multitude of tutorials and examples that only show the loss decreasing graph but the actual results are not evaluated (and thus might have some glaring defects).

Re: A Kaggle Grandmaster cheated in $25k AI contest with hidden code

#110

Post by alanfranz »

laydn wrote: post #109

Slightly off topic: The nature of the competition is a bit worrying to me. They're essentially letting an algorithm decide which dogs have the best chance of adoption and which to euthanize, aren't they?

Are you sure? Aren't they just trying to show the most appropriate pets to the most appropriate people in order to make adoptions quicker? Where do you read the euthanization part?
Post reply on HN