How Khan Academy is using machine learning to assess student mastery • Hacker News Board

Re: How Khan Academy is using machine learning to assess student mastery

#11

Post by webspiderus » Wed, Nov 02, 2011, 6:59 PM UTC

lliiffee wrote: post #4

I think we should note that logistic regression has been around forever, and should probably be considered property of "statistics", not "machine learning".

this is literally the first thing taught in both of the machine learning classes I've taken from Prof. Ng at Stanford, so maybe it's the application of logistic regression more than the estimation technique itself that makes machine learning?

Re: How Khan Academy is using machine learning to assess student mastery

#12

Post by danteembermage » Wed, Nov 02, 2011, 7:12 PM UTC

lliiffee wrote: post #4

I think we should note that logistic regression has been around forever, and should probably be considered property of "statistics", not "machine learning".

Iteratively solving for the model parameters using gradient descent is not standard practice in a statistics class; paying attention to the numerical methods behind logistic regression is a very CS kind of thing to do.

Re: How Khan Academy is using machine learning to assess student mastery

#13

Post by lliiffee » Wed, Nov 02, 2011, 7:13 PM UTC

webspiderus wrote: post #11

lliiffee wrote: post #4

I think we should note that logistic regression has been around forever, and should probably be considered property of "statistics", not "machine learning".

this is literally the first thing taught in both of the machine learning classes I've taken from Prof. Ng at Stanford, so maybe it's the application of logistic regression more than the estimation technique itself that makes machine learning?

I think it's more that machine learning builds off of earlier work in statistics. If I remember correctly, that class discusses lots of other things that are used in ML but were invented elsewhere (gradient descent, maximum likelihood, matrix methods, etc.)

Arguably this is all just semantics (nature knows no stats/ML divide), but as a ML person I know this drives statistics folks crazy.

Re: How Khan Academy is using machine learning to assess student mastery

#14

Post by noelwelsh » Wed, Nov 02, 2011, 7:20 PM UTC

You should look at liblinear and Vowpal Wabbit. The former gives you a super-fast regularized logistic regression (and many other things). The later gives you online classification, and you can fake a probabilistic model.

On a related note, you're wasting clicks using A/B testing. I emailed you guys about using a better online method (a bandit algorithm) but never heard back from anyone. If that's of interest, drop me a line (noel at untyped dot com).

Update It's occurred to me that you're using GAE, and so probably can't run C libraries like the above two projects. There is a Java library here: http://code.google.com/p/boxer-bayesian-regression/ If you're going to do per user and per exercise models you'll have many fewer data points to train your models on. You should consider sharing data between models or use a model that will give some measure of uncertainty in it's predictions. The Bayesian LR code I referenced above will give some measure of uncertainty. There is a stack of (really interesting) work on other methods that will also do this.

Re: How Khan Academy is using machine learning to assess student mastery

#15

Post by brown9-2 » Wed, Nov 02, 2011, 7:36 PM UTC

gtrak wrote: post #8

pnewhook wrote: post #5

Wow, this is literally the exact application of the Stanford Machine Learning class up to this point.

the professor did say people make careers off of what we already know with linear and logistic regression.

I loved how giddy he was in the class videos when he stated that.

Re: How Khan Academy is using machine learning to assess student mastery

#16

Post by brown9-2 » Wed, Nov 02, 2011, 7:39 PM UTC

noelwelsh wrote: post #14

You should look at liblinear and Vowpal Wabbit. The former gives you a super-fast regularized logistic regression (and many other things). The later gives you online classification, and you can fake a probabilistic model. On a related note, you're wasting clicks using A/B testing. I emailed you guys about using a better online method (a bandit algorithm) but never heard back from anyone. If that's of interest, drop m…

I have nothing to do with KA but I am curious by what you mean about A/B testing and bandit algorithms. Would you mind elaborating or sharing a link or two?

Re: How Khan Academy is using machine learning to assess student mastery

#17

Post by brendano » Wed, Nov 02, 2011, 7:41 PM UTC

webspiderus wrote: post #11

lliiffee wrote: post #4

I think we should note that logistic regression has been around forever, and should probably be considered property of "statistics", not "machine learning".

this is literally the first thing taught in both of the machine learning classes I've taken from Prof. Ng at Stanford, so maybe it's the application of logistic regression more than the estimation technique itself that makes machine learning?

There used to be an earlier era of machine learning that wasn't as statistical. Ng, and most other current ML researchers, now heavily draw on mainstream statistics. It really does make sense to do logistic regression as the foundation for later stuff.

The terminology confusions, I think, stems from the earlier era of ML research.

Re: How Khan Academy is using machine learning to assess student mastery

#18

Post by noelwelsh » Wed, Nov 02, 2011, 7:42 PM UTC

brown9-2 wrote: post #16

noelwelsh wrote: post #14

You should look at liblinear and Vowpal Wabbit. The former gives you a super-fast regularized logistic regression (and many other things). The later gives you online classification, and you can fake a probabilistic model. On a related note, you're wasting clicks using A/B testing. I emailed you guys about using a better online method (a bandit algorithm) but never heard back from anyone. If that's of interest, drop m…

I have nothing to do with KA but I am curious by what you mean about A/B testing and bandit algorithms. Would you mind elaborating or sharing a link or two?

I'm totally pimping my own warez here:

http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m...

http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html

Re: How Khan Academy is using machine learning to assess student mastery

#19

Post by Eliezer » Wed, Nov 02, 2011, 8:35 PM UTC

noelwelsh wrote: post #18

brown9-2 wrote: post #16

Earlier quoted context omitted.

I have nothing to do with KA but I am curious by what you mean about A/B testing and bandit algorithms. Would you mind elaborating or sharing a link or two?

I'm totally pimping my own warez here: http://untyped.com/untyping/2011/02/11/stop-ab-testing-and-m... http://www.mynaweb.com/blog/2011/09/13/myna-vs-ab.html

(Reads links.) I've been going around telling people for a while that A/B testing is non-Bayesian but I didn't realize there was an off-the-shelf solution! You need to pimp your wares more often.

Re: How Khan Academy is using machine learning to assess student mastery

#20

Post by Eliezer » Wed, Nov 02, 2011, 8:37 PM UTC

Good stuff, but the thought occurs to me that what you really want to know is when the user has stopped learning - i.e., when proficiency stops increasing as a result of doing more problems - and how much each individual problem increases proficiency. But that would undoubtedly be more complicated.