Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stack Exchange Machine Learning Contest

blog.stackoverflow.com

Post reply on HN
Print view View original

31–40 of 67 posts

Re: Stack Exchange Machine Learning Contest

#31

Post by sirtaptap »

kmontrose wrote: post #5
teh_klev wrote: post #4

Earlier quoted context omitted.

Does every Stack Exchange post need to be hijacked by yet another whine about site policy and moderation activities?

It's in the terms of service. But seriously, Stack Exchange thrives because of its focus (which is maintained in large part by its culture of moderation, though only a small part of that is done by actual "moderators"; the community self-polices pretty well). This contest is aimed at catching posts that would be closed before they're posted. Ideally we'll get a classifier that lets us give guidance to the askers. Bet…

Yeah, I figured this would be used to possibly provide some Just In Time help. Can't say I'm surprised people read that as "people posting questions detected to be closed will be shot in the face and fed to the SE Inc pet Unicorn"

Re: Stack Exchange Machine Learning Contest

#32

Post by rm999 »

Homunculiheaded wrote: post #9
rm999 wrote: post #2

Looks like a cool contest, I may check it out. What bothers me about modeling contests (I've taken part in several, it's my field) is they often reward putting 90% of your effort into extracting relatively small performance gains. For one thing it's not a realistic operating environment, there are usually many other factors more important than pure performance like upkeep, cost, speed, etc. This is why the netflix co…

As someone who went from top 5 to somewhere in the 60s in one contest, and reviewing results of past contests, I believe a lot of those small tweaks for slight gains in leader board scores end up penalizing the contestant for over-fitting. I saw a similar complaint to yours in a couple of forums but I do believe more often than not those small performance gains in the leader board actually hurt final scores. Addition…

It's not about small tweaks, it can be substantial additions to a model that improve its actual, out-of-sample performance. A popular method in these contests is ensembling, which involves building many sub-models and combining their scores into a single ensemble model. The netflix winner used ~100 sub-models in their ensemble, but the vast majority of the predictive power came from just three of those sub-models (can't find the source now).

Re: Stack Exchange Machine Learning Contest

#33

Post by MaysonL »

Stefan_H wrote: post #26
pc86 wrote: post #20

Earlier quoted context omitted.

> I guess, but Zookeepers could also potentially talk about "What are the pros and cons of Gorillas vs Sharks?" And on an appropriate forum with as many readers as SO/SE you'd likely get well-thought out responses going far into detail regarding purchase cost, habitat maintenance, prevalence of skilled keepers, etc. Someone who actually was deciding whether to add a Gorilla enclosure or a shark aquarium would find it…

You hit the nail on the head... "an appropriate forum with as many readers as SO/SE" Stack overflow on the other hand is for people to: [A]sk practical, answerable questions based on actual problems that you face. Chatty, open-ended questions diminish the usefulness of our site and push other questions off the front page.

* Chatty, open-ended questions diminish the usefulness of our site*

How, pray tell?

Re: Stack Exchange Machine Learning Contest

#34

Post by Homunculiheaded »

mej10 wrote: post #30
Homunculiheaded wrote: post #23

Earlier quoted context omitted.

If you read through the past winners you'll find that in many cases a very simple model will win. I believe one of the winners that posted a blog post had pretty much the background you describe. When I started I was in a similar position to you and just wanted to see if I could even tread water with some of the really knowledgeable members of the community. I ended up placing in the top 5 for one of the contests I w…

Awesome, thanks for the info. I am checking out some of the benchmarks now. Why do you think it is that simple models often win? Is it due to the experts no participating or is there a lot more low-hanging fruit than I previously thought? Or just that simple models are easier to use and reason with for humans and thus easier to get right.

If you look at some of the bios of the top 50 kaggler's there's some pretty impressive backgrounds there, and they participate heavily. So I don't think that's the reason.

I know for my own beginner mistakes, it's a big error to try something out-of-the-box and immediately try to get better cv scores by creating much more complicated solutions.

The truth is a lot of work has been put into any standard implementation of an SVM, RandomForest etc (and even more work has been put into the theory behind those algorithms). Since I haven't come in 1st in any competition and am not a ML expert I don't think I can give you the correct strategy to win. But I can say as a general trend, all of my attempts to create non-standard complicated models did terribly, and many of the decisions I made based on research into fixing a particular problem in a known solution seemed to be better performing (i.e. "How to deal with imbalanced classification problems with a RF?" type questions)

Re: Stack Exchange Machine Learning Contest

#35

Post by teh_klev »

Bamafan wrote: post #28
cletus wrote: post #3

It's nice to see this kind of contest but the topic just sets me off on a much-needed rant. The moderator situation on Stackoverflow is getting out of control. I see a Q&A site as having three main groups: 1. People who ask questions; 2. People who answer questions; and 3. People who edit/moderate questions. Even 2+ years ago there was a lot of lip service paid to the value of (3). I disagreed then and it's only been…

"Take this question from Stackoverflow: Database development mistakes made by application developers [3], a question I spent some time answering and that people seemed to appreciate the answer to (based on comments and 1000+ upvotes). It is closed as "not constructive". This is hardly a unique phenomenon. We've all seen many interesting questions posted here that are now closed or locked and who knows how many have b…

"They had a preconceived notion of what StackOverflow Is and they are going to stick to it, users be damned."

Which is why the site is hugely successful. If you want discussion then do it here or on Reddit. I don't want the questions I ask about why something doesn't work in jQuery or C# or C++ or whatever drowned out by Ruby Vs Python posts or stuff like the question cletus mentioned above, or "what have I got in my pocket" mysteries.

The brilliance of Stack Overflow is how quickly one can get answers to "specific programming problems" because everyone on the site is focused on answering these types of questions, not participating in discussion and navel gazing.

Re: Stack Exchange Machine Learning Contest

#36

Post by numlocked »

mej10 wrote: post #30
Homunculiheaded wrote: post #23

Earlier quoted context omitted.

If you read through the past winners you'll find that in many cases a very simple model will win. I believe one of the winners that posted a blog post had pretty much the background you describe. When I started I was in a similar position to you and just wanted to see if I could even tread water with some of the really knowledgeable members of the community. I ended up placing in the top 5 for one of the contests I w…

Awesome, thanks for the info. I am checking out some of the benchmarks now. Why do you think it is that simple models often win? Is it due to the experts no participating or is there a lot more low-hanging fruit than I previously thought? Or just that simple models are easier to use and reason with for humans and thus easier to get right.

I work at Kaggle.

In many cases where simple models win, there's some insight into the data that the winner found - engineered a new feature, or noticed a pattern and appropriately tuned a particular method. Where those insights exist, they often overshadow any gains by super-sophisticated ML techniques.

Re: Stack Exchange Machine Learning Contest

#37

Post by usea »

ricardobeat wrote: post #21

probabilityOfClosing = (question) -> text = question.text.toLowerCase() return (text.length / (text.indexOf('jquery') + 2)) / 100

I realize this was in jest, but your algorithm says that the shorter a post, the less likely it is to be closed. A post of zero length has a 0% chance of being closed.

Also, the result is not bound to 0..1

Re: Stack Exchange Machine Learning Contest

#38

Post by mej10 »

numlocked wrote: post #36
mej10 wrote: post #30

Earlier quoted context omitted.

Awesome, thanks for the info. I am checking out some of the benchmarks now. Why do you think it is that simple models often win? Is it due to the experts no participating or is there a lot more low-hanging fruit than I previously thought? Or just that simple models are easier to use and reason with for humans and thus easier to get right.

I work at Kaggle. In many cases where simple models win, there's some insight into the data that the winner found - engineered a new feature, or noticed a pattern and appropriately tuned a particular method. Where those insights exist, they often overshadow any gains by super-sophisticated ML techniques.

Makes sense. Thanks!

Re: Stack Exchange Machine Learning Contest

#39

Post by kmontrose »

m_myers wrote: post #25
bnegreve wrote: post #18

Earlier quoted context omitted.

This particular post is about replacing human moderators with bots, whining about SE moderation policy is actually quite related to the post.

Replacing moderators with bots would be a terrible idea and hopefully will never happen. Kevin Montrose clarified it in a comment: > To be clear, we’re not intending to have an algorithm automatically close posts. This is aimed at improving our quality metrics (which guide users before posting) and auto-flagging facilities (which help focus moderation attention).

There is a 0% chance of replacing moderators with bots.

I don't even know how you could read that blog post and come to that conclusion...

Re: Stack Exchange Machine Learning Contest

#40

Post by smackfu »

kmontrose wrote: post #5
teh_klev wrote: post #4

Earlier quoted context omitted.

Does every Stack Exchange post need to be hijacked by yet another whine about site policy and moderation activities?

It's in the terms of service. But seriously, Stack Exchange thrives because of its focus (which is maintained in large part by its culture of moderation, though only a small part of that is done by actual "moderators"; the community self-polices pretty well). This contest is aimed at catching posts that would be closed before they're posted. Ideally we'll get a classifier that lets us give guidance to the askers. Bet…

Good luck with that. Even your current human moderators can be extremely frustrating to deal with. Can't wait until they have the force of an algorithm behind them.
Post reply on HN