AI competitions don’t produce useful models • Hacker News Board

Re: AI competitions don’t produce useful models

#61

Post by prepend » Mon, Sep 23, 2019, 7:00 AM UTC

Not AI related, but I had a boss who thought hackathons were great because they would spontaneously produce great things in a day.

I’m kind of an introvert so I hate hackathons but did one and had about 75 code for a day. My boss wasn’t happy but I was surprised at the fun culture and connections created across teams. Five years later people mention something they learned or a technique they use or someone they met that they continue to work with.

I don’t think these competitions are the best mean for solving specific problems, though think they are part of a good portfolio.

I think that they are valuable for other purposes like communication and interoperability.

Re: AI competitions don’t produce useful models

#62

Post by kqr » Mon, Sep 23, 2019, 10:12 AM UTC

m0zg wrote: post #58

kqr wrote: post #56

Earlier quoted context omitted.

> How do you "overfit" on the test set if you don't have the test set? By being lucky. And with a large enough number of solutions, at least one solution will almost surely be lucky.

I guess I'm extraordinarily "lucky" then. I don't participate in competitions, but I'm often "luckier" than entire teams of people working tirelessly for months. And I charge a lot of money for it.

I'm intentionally misunderstanding your comment now for academic reasons: it's great that you recognise the role of survival bias and luck in becoming successful at many things.

Besides that, I'm very curious about who you are and what you do. Willing to reveal more?

Re: AI competitions don’t produce useful models

#63

Post by probinso » Tue, Sep 24, 2019, 12:24 AM UTC

sgt101 wrote: post #43

probinso wrote: post #30

Our company is very thankful for the models developed for a competition we held through kaggle. The model is in our current release product, and is the current known state of the art for it's task in accuracy / performance.

How sure are you that it works? I know that this sounds really flippant, but I have seen a number of well regarded models that when we / I took them to bits were discovered to actually not be doing better than dominant class prediction, and in a couple of cases had actually become arbitrary. When did someone last really pick it apart in its operational environment?

yeah, the model works. Its individual identification task there are many individuals, and we have lots of labeled data that we held out from the competition.

Re: AI competitions don’t produce useful models

#64

Post by probinso » Tue, Sep 24, 2019, 12:28 AM UTC

adilmoujahid wrote: post #41

probinso wrote: post #30

Our company is very thankful for the models developed for a competition we held through kaggle. The model is in our current release product, and is the current known state of the art for it's task in accuracy / performance.

Is it a single model or stacking of multiple models?

There were several top contenders that did ensemble. We went with a single model (also in top few) for reasons of integration cost. I Don't actually remember which placed first for the competition.