Live data from Hacker News

Machine learning isn't Kaggle competitions

jvns.ca

11–20 of 24 posts

Re: Machine learning isn't Kaggle competitions

#11
Hm, looks like this argument about Kaggle pops up all the time.

This is true, that getting and cleaning data is an important part of a workflow in data science and it's a necessary skill to have. But I don't believe anyone ever claimed Kaggle is enough to become a good data scientist. That's like studying algorithms is very useful, but who said it's enough to become good at software engineering? The skill consist of multiple parts which can be practiced separately or in combination in a real project where you do everything from scratch.

Re: Machine learning isn't Kaggle competitions

#12
There is too much hype, solving data-related problems requires many different skills and it depends on the problem. In general you need background in quite a few of these areas linear algebra, probability, statistical modeling, stochastic processes, computer science, programming, signal processing, etc. We used to call this electrical/computer engineering :-)

Re: Machine learning isn't Kaggle competitions

#13
post #11

Hm, looks like this argument about Kaggle pops up all the time. This is true, that getting and cleaning data is an important part of a workflow in data science and it's a necessary skill to have. But I don't believe anyone ever claimed Kaggle is enough to become a good data scientist. That's like studying algorithms is very useful, but who said it's enough to become good at software engineering? The skill consist of…

It's not as straightforward as Kaggle representing one stage in a pipeline. Kaggle also fixes this stage in an unnatural way.

In real life, there is a feedback loop from modelling, to data processing. In Kaggle all you can do is improve your model. You can't go back and change how data was processed, or collect different sorts of data.

Also, some flows simply don't fit into Kaggle's train/test paradigm. E.g. suppose you have an online algorithm that continuously updates parameters. Or situations where the model is used to generate the train/test data.

Finally some situations may have stringent computational requirements, e.g. <100ms to classify a single instance.

Re: Machine learning isn't Kaggle competitions

#14
post #10

"Once I played against an NBA player and I couldn't do anything against him. I felt sad and demoralized. I still don't think I could win against an NBA player, but everyday I do (among other things) play basketball with my friends! And, you know, there're so many problems in basketball aside of dribbling and stuff: you have to find space to play with your friends, you have to convince the guy that doesn't want to pla…

To me it sounded more like "the problems they do at Mathematical Olympiads are not the real mathematics done at academia/the enterprise," which turns out to be quite true in many ways.

Re: Machine learning isn't Kaggle competitions

#16
Why Machine Learning is Kaggle competitions.

  I used an out-of-the-box algorithm, messed around a bit, 
  and definitely did not make the leaderboard.
Because that is not Kaggle competitions. Nearly everyone on the leaderboard is proficient in data analysis and machine learning. They all tried that out-of-the-box algorithm for their attempt. But they did not give up so easily.

  Understand the business problem
  If you want to predict flight arrival times, what are 
  you really trying to do?
This is not different from Kaggle competitions, this is a tip for performing better in Kaggle competitions. See also the GE Flight Quest: https://www.gequest.com/c/flight Those winners used industry-standard machine learning, optimization techniques, but also creative insights and hunches, like tweaking the target labels:

"A next step is to ask, “What should I actually be predicting?”. This is an important step that is often missed by many – they just throw the raw dependent variable into their favorite algorithm and hope for the best. But sometimes you want to create a derived dependent variable. I’ll use the GE Flight Quest as an example: you don't want to predict the actual time the airplane will land; you want to predict the length of the flight; and maybe the best way to do that is to use the ratio of how long the flight actually was to how long it was originally estimated to be and then multiply that times the original estimate." - Steve Donoho - http://blog.kaggle.com/2014/08/01/learning-from-the-best/

Furthermore, it is entirely clear to everyone that data science in a business setting and in a competitive sport setting is different. To say they are equal, would be to say something like: paintball is equal to being in the military. But to say that Kaggle is not machine learning is to say: paintball requires no marksmanship.

There are some very messy, unwieldy datasets on Kaggle right now. For example the Seizure Detection challenge has many GBs of raw sensor data, from just a few patients. This would require a competitor to clean, understand problem domain, understand evaluation metrics, measure cross validation and put your model in production on your laptop in the evening hours.

The author of that blogpost is invited to team up, with me or others. Let's see if we can use machine learning to improve some pressing issues. I'd also love it if Stripe can host a contest on Kaggle.

Re: Machine learning isn't Kaggle competitions

#17
post #11

Hm, looks like this argument about Kaggle pops up all the time. This is true, that getting and cleaning data is an important part of a workflow in data science and it's a necessary skill to have. But I don't believe anyone ever claimed Kaggle is enough to become a good data scientist. That's like studying algorithms is very useful, but who said it's enough to become good at software engineering? The skill consist of…

It's not as straightforward as Kaggle representing one stage in a pipeline. Kaggle also fixes this stage in an unnatural way. In real life, there is a feedback loop from modelling, to data processing. In Kaggle all you can do is improve your model. You can't go back and change how data was processed, or collect different sorts of data. Also, some flows simply don't fit into Kaggle's train/test paradigm. E.g. suppose…

Sure, everything "in real life" is more complicated. I think Kaggle to data science is as programming contests to Software Engineering. Will you say that doing programming challenges is "unnatural"? Is it useless or bad? Because you may also argue that you never have to solve problems for speed, you rarely see clearly-defined problems, you rarely get to apply complicated or rare algorithms in real life situation.

The thing is Kaggle helps developing certain skills. It doesn't mean it helps you developing all necessary skills for your "real life". Your real life and your job may require unique combination of skills and the only way to fully prepare for it is to actually do real project that are specific to your job.

Re: Machine learning isn't Kaggle competitions

#18
post #10

"Once I played against an NBA player and I couldn't do anything against him. I felt sad and demoralized. I still don't think I could win against an NBA player, but everyday I do (among other things) play basketball with my friends! And, you know, there're so many problems in basketball aside of dribbling and stuff: you have to find space to play with your friends, you have to convince the guy that doesn't want to pla…

To me it sounded more like "the problems they do at Mathematical Olympiads are not the real mathematics done at academia/the enterprise," which turns out to be quite true in many ways.

Not just math -- even the problems done at programming contests have fairly little to do with programming!

I've worked at Google for nearly 10 years -- not sure I can solve the Google code Jams :) Or at least I haven't been motivated to do so. I know all my basic data structures and algorithms, but the questions don't seem that motivating.

Re: Machine learning isn't Kaggle competitions

#19

Why Machine Learning is Kaggle competitions. I used an out-of-the-box algorithm, messed around a bit, and definitely did not make the leaderboard. Because that is not Kaggle competitions. Nearly everyone on the leaderboard is proficient in data analysis and machine learning. They all tried that out-of-the-box algorithm for their attempt. But they did not give up so easily. Understand the business problem If you want…

Completely agree. Everything that is supposedly not addressed in Kaggle actually is, aside from productionizing and monitoring your model in production. Sure it's a simplified less open ended version of what you'll encounter in the real world, but that doesn't mean that many of the core concepts don't translate. It's kind of like saying that you shouldn't do your calculus practice problems, because math in the real world is never so straightforward.

I think the most legitimate knock against Kaggle is that in many business settings there probably isn't much value in improving that extra .00001 (but obviously there are exceptions).

Anyway, I think that that even someone very experienced in machine learning would learn something doing a Kaggle competition, especially if the competition is in an area outside their core expertise.

Re: Machine learning isn't Kaggle competitions

#20
post #10

"Once I played against an NBA player and I couldn't do anything against him. I felt sad and demoralized. I still don't think I could win against an NBA player, but everyday I do (among other things) play basketball with my friends! And, you know, there're so many problems in basketball aside of dribbling and stuff: you have to find space to play with your friends, you have to convince the guy that doesn't want to pla…

To me it sounded more like "the problems they do at Mathematical Olympiads are not the real mathematics done at academia/the enterprise," which turns out to be quite true in many ways.

I'd rather say that mathematics at academia aren't real mathematics. It pretty much depends on academia, of course, but what is generally taught to students under the name of "mathematics" (especially on technical programs, like engineering, CS, etc.) has less mathematical value than typical problem at Olympiads. But that wasn't my point, anyway.

Yes, that is true that "pure something" is almost always significantly different from "something in the real life". That's pretty obvious, actually. I guess that there's virtually nothing in the "real life" that would consist from only one "core" activity. In fact, it's the opposite: you make some activity "pure" in order to master it, because in the real life there's so much that distracts you from mastering that one essential skill, whatever would it be. Plubmers and painters (and almost everyone else) must to be able to talk to others and to solve totally human-related problems in order to do their job. Person you would call sniper actually does much more running, crawling, waiting and hiding than shooting. Or does it really surprise anyone that policemen do tons of paperwork every day and not so much of that stuff they show in the movies? And somewhat opposite example: it's obvious that cook's work isn't only slicing vegetables, but have you ever seen how any real cook handles his knife? Please do, it's sight to behold. There's nothing "pure" in this world, no surprise here.

So if it would be Yan LeCun writing something like this it would be justifiable (although I would be rather surprised to hear that), but in this particular case it's more like saying "Kaggle isn't important, data science is me!", in which case I find that NBA example to be OK.

To summarise: "data science is more than building a model" — yes, "Machine learning isn't Kaggle competitions" — no.

Post reply on HN