Stack Exchange Machine Learning Contest
blog.stackoverflow.com
Stack Exchange Machine Learning Contest
1–10 of 67 posts
Re: Stack Exchange Machine Learning Contest
#2Looks like a cool contest, I may check it out. What bothers me about modeling contests (I've taken part in several, it's my field) is they often reward putting 90% of your effort into extracting relatively small performance gains. For one thing it's not a realistic operating environment, there are usually many other factors more important than pure performance like upkeep, cost, speed, etc. This is why the netflix contest winning models couldn't go into production. The other issue I have is that people with other commitments (like a job) don't really stand a chance, it's usually very time-consuming to go from fifth place to first.
Re: Stack Exchange Machine Learning Contest
#3It's nice to see this kind of contest but the topic just sets me off on a much-needed rant.
The moderator situation on Stackoverflow is getting out of control. I see a Q&A site as having three main groups:
1. People who ask questions;
2. People who answer questions; and
3. People who edit/moderate questions.
Even 2+ years ago there was a lot of lip service paid to the value of (3). I disagreed then and it's only been reaffirmed by subsequent events. To be clear: it's not that I think these functions have no value, it's that they are, at best, secondary to content creation.
The problem is that these roles without diligent oversight attract the wrong kinds of people (eg [1] [2] and a scandal a few years about an admin black list that I can't seem to find right now).
Take this question from Stackoverflow: Database development mistakes made by application developers [3], a question I spent some time answering and that people seemed to appreciate the answer to (based on comments and 1000+ upvotes). It is closed as "not constructive". This is hardly a unique phenomenon. We've all seen many interesting questions posted here that are now closed or locked and who knows how many have been deleted.
The kind of person you end up is overly pedantic and a real stickler for an arbitrary set of rules.
Editors/moderators are the bureaucrats of the Internet.
As Oscar Wilde said, “The bureaucracy is expanding to meet the needs of the expanding bureaucracy.” [4]. These sorts of people just invent work for themselves in the absence of anything to do.
Joel needs to make some changes to Stackoverflow. It's rapidly going the way of the old Usenet days when anything interesting gets shot down and anything else gets closed and the OP lambasted for not having found the 17 previous duplicates. Not good.
The biggest problem I see is an extreme interpretation of what is "subjective". "What language should I learn?" is an obviously subjective question. In the absence of any concrete criteria, it's hard to give a useful answer.
But consider a question like "What are the pros and cons of Sinatra vs Rails?" This sort of question (IMHO) absolutely has value as someone experienced with both could enumerate the relative merits of each in a pretty objective fashion without making an absolute determination. This is something that absolutely could have value to anyone evaluating Ruby Web frameworks.
So, back to this post, what are the odds of any particular question being closed? it seems to be positively correlated with how much time has passed (since SO's inception) and how interesting the question is.
[1]: http://www.nbcnews.com/technology/technolog/wikipedia-admins...
[2]: http://www.searchenginepeople.com/blog/most-notorious-wikipe...
[3]: http://stackoverflow.com/questions/621884/database-developme...
[4]: http://www.goodreads.com/quotes/130452-the-bureaucracy-is-ex...
Re: Stack Exchange Machine Learning Contest
#4It's nice to see this kind of contest but the topic just sets me off on a much-needed rant. The moderator situation on Stackoverflow is getting out of control. I see a Q&A site as having three main groups: 1. People who ask questions; 2. People who answer questions; and 3. People who edit/moderate questions. Even 2+ years ago there was a lot of lip service paid to the value of (3). I disagreed then and it's only been…
Does every Stack Exchange post need to be hijacked by yet another whine about site policy and moderation activities?
Re: Stack Exchange Machine Learning Contest
#5It's nice to see this kind of contest but the topic just sets me off on a much-needed rant. The moderator situation on Stackoverflow is getting out of control. I see a Q&A site as having three main groups: 1. People who ask questions; 2. People who answer questions; and 3. People who edit/moderate questions. Even 2+ years ago there was a lot of lip service paid to the value of (3). I disagreed then and it's only been…
Does every Stack Exchange post need to be hijacked by yet another whine about site policy and moderation activities?
It's in the terms of service.
But seriously, Stack Exchange thrives because of its focus (which is maintained in large part by its culture of moderation, though only a small part of that is done by actual "moderators"; the community self-polices pretty well).
This contest is aimed at catching posts that would be closed before they're posted. Ideally we'll get a classifier that lets us give guidance to the askers. Better education of new/confused/troublesome users, leading to better posts, and fewer closed questions overall.
(Disclaimer: SE Inc. employee, yadda yadda yadda)
Re: Stack Exchange Machine Learning Contest
#6If you're someone who's interested in ML/datamining but haven't had a chance to put your ideas to any hard/interesting problems I strongly recommend a kaggle contest. It's one thing to plug some data into a random forest and go "oh cool, I guess that did okay" and entirely another to see how other competitors are comparing.
one of the biggest challenges I've found in implementing ML projects is I don't have a great sense of when I've really gotten the most info out of the data. I'm not particularly competitive but the contest format is great for this. When you see that a solution you'd normally be happy with ranks in the lower half of the answer you're really pushed to improve your solution.
This is leads you to learn your tools and algorithms better. For a couple of contests I took seriously I ended up learning tons about R, spent most of my nights reading academic papers on various newer techniques, and also read through a few books. On top of all that you really should spend time reading up on how past winner have won which gives a bunch of practical insight into approaching different ML problems.
In one contest I tried the hardest in I actually placed terribly after the final results were calculated, but looking over what went wrong I was amazed to see that I actually did progress really far with my understanding of ml. I'd say a month of seriously competing is easily worth a semester long grad class.
Re: Stack Exchange Machine Learning Contest
#7Amusing side note: I clicked on their job ad, apparently they score 10/12 on the "Joel Test", which according to the link indicates "serious problems".
Re: Stack Exchange Machine Learning Contest
#8It's nice to see this kind of contest but the topic just sets me off on a much-needed rant. The moderator situation on Stackoverflow is getting out of control. I see a Q&A site as having three main groups: 1. People who ask questions; 2. People who answer questions; and 3. People who edit/moderate questions. Even 2+ years ago there was a lot of lip service paid to the value of (3). I disagreed then and it's only been…
Does every Stack Exchange post need to be hijacked by yet another whine about site policy and moderation activities?
No, but it's particularly apposite on this post, as SE is asking for a machine learning algorithm is to assist in moderating questions:
an algorithm that predicts whether (and for what reason) a question will be closed.
which raises all sorts of questions as to what the reasons are for which questions [should][1] be closed - this is a grey area, and there is much argument on SE itself over what sort of questions are considered acceptable, to the extent that they have started retroactively disabling lots of content which doesn't fit with an arbitrary set of rules about what a Q&A site should have on it. Without some moderation obviously the site would descend into chaos, but with very heavy handed and arbitrary moderation it will atrophy and the very people creating the quality content they want will leave, leaving them with moderators (robot or not), new users and trolls, and not much else.
Personally I think it's a huge mistake for SE to start banning questions on the basis of them being not constructive. All their other categories of problem questions make sense, but trying to ban questions that are too controversial or involve opinions is IMHO unwise - that's exactly the sort of question which leads to engaging content on SO, even if some of it verges on a troll. This has been the problematic area for them and has lead to them marking lots of useful posts as not constructive even though they clearly are constructive and informative, just because they fall on the wrong side of a line decided retrospectively to declare certain questions unquestions, and others valuable.
It will be very interesting to see if any of their robot moderators are useful in delineating this more problematic area of questions which are controversial or involve opinions - potentially that is every question/answer set more complex than 'what is 2 + 2?', and that line can vary dramatically depending on the moderator, and their opinions.
So I think this raises an interesting (though perennial) question about how heavy handed community moderation should be; in some ways related to those questions raised recently about quality and moderation on HN.
Re: Stack Exchange Machine Learning Contest
#9Looks like a cool contest, I may check it out. What bothers me about modeling contests (I've taken part in several, it's my field) is they often reward putting 90% of your effort into extracting relatively small performance gains. For one thing it's not a realistic operating environment, there are usually many other factors more important than pure performance like upkeep, cost, speed, etc. This is why the netflix co…
As someone who went from top 5 to somewhere in the 60s in one contest, and reviewing results of past contests, I believe a lot of those small tweaks for slight gains in leader board scores end up penalizing the contestant for over-fitting. I saw a similar complaint to yours in a couple of forums but I do believe more often than not those small performance gains in the leader board actually hurt final scores.
Additionally for contests like Heritage Health [0], I believe the necessary goal of RMSLE of less than 0.4 is not considered possible (I came across this in the forums but never verified), so even if the contestants just inch past 0.4 it would still be something impressive.
Re: Stack Exchange Machine Learning Contest
#10It's nice to see this kind of contest but the topic just sets me off on a much-needed rant. The moderator situation on Stackoverflow is getting out of control. I see a Q&A site as having three main groups: 1. People who ask questions; 2. People who answer questions; and 3. People who edit/moderate questions. Even 2+ years ago there was a lot of lip service paid to the value of (3). I disagreed then and it's only been…
[deleted]