How Khan Academy is using machine learning to assess student mastery • Hacker News Board

Re: How Khan Academy is using machine learning to assess student mastery

#41

Post by ced » Thu, Nov 03, 2011, 8:16 PM UTC

xenophanes wrote: post #35

ced wrote: post #32

Earlier quoted context omitted.

It's getting too late for me to think about statistics :) A few points: Similarly, imagine a study of coins which had a stopping rule to stop whenever you have at least 60% heads. You'll always be able to get that result and conclude the coin is biased, even if all coins used are fair. This is not true. Because of the law of large numbers, the probability of ever reaching the 60% decreases with time. I do think the o…

FYI I edited my post to mention the issue about the coins (your first point) shortly after submitting it. I'm guessing you read the non-edited version. > Proper Bayesian result reporting doesn't say "We believe that the coin is biased". We would rather say "The probability that this coin is biased is 60%, subject to our assumptions and model". I'm not really sure what you're getting at here. None of the coins are bia…

xenophanes,

I've been thinking about the problem a lot today. I'm pretty sure that my point is basically right, if the model is correct, but my ideas are not clear enough to explain it properly. Model correctness in Bayesian statistics is a complicated problem, and as far as I can tell, it's not a completely solved one. Bayesians usually agree about their calculations, but there's heavy debate about the "philosophy".

In any case, maybe you'll find Eliezer's other post insightful:

http://lesswrong.com/lw/1gc/frequentist_statistics_are_frequ...

I really hope to figure out model correctness, and this optional stopping problem looks a good vector of attack.

Thank you for the discussion, and sorry for leaving you hanging!

Cedric

(if there's any Bayesian out there willing to continue the discussion, my email is in my profile)

Re: How Khan Academy is using machine learning to assess student mastery

#42

Post by ced » Thu, Nov 03, 2011, 8:46 PM UTC

equark wrote: post #33

xenophanes wrote: post #30

Earlier quoted context omitted.

Oh, I see what you mean about fixed . It's hard to generalize about all data-dependent stopping rules. I do think the one that says "If I get one conclusion, stop. If I get the other, keep trying," has got to be a bad idea! It prevents you from possibly finishing an experiment that concludes you're wrong. The possibility of not terminating seems especially problematic. But if the rule was more like "Roll 10 dice. If…

You're stuck in frequentist thinking. "Bias" is a property of repeated sampling -- the expectation over repeated samples. But we just have one! The relevant question is what is your best guess for p, the probability the coin will be heads. Under a uniform prior [0, 1] the posterior mean is the empirical mean. How you sample is of no consequence. The likelihood/posterior f(p|#heads, #tails) is p^(#heads)(1-p)^(#tails)…

Off-topic, but what happened to statsia? I was curious to see what you were working on.

Re: How Khan Academy is using machine learning to assess student mastery

#43

Post by xenophanes » Thu, Nov 03, 2011, 9:24 PM UTC

ced wrote: post #41

xenophanes wrote: post #35

Earlier quoted context omitted.

FYI I edited my post to mention the issue about the coins (your first point) shortly after submitting it. I'm guessing you read the non-edited version. > Proper Bayesian result reporting doesn't say "We believe that the coin is biased". We would rather say "The probability that this coin is biased is 60%, subject to our assumptions and model". I'm not really sure what you're getting at here. None of the coins are bia…

xenophanes, I've been thinking about the problem a lot today. I'm pretty sure that my point is basically right, if the model is correct, but my ideas are not clear enough to explain it properly. Model correctness in Bayesian statistics is a complicated problem, and as far as I can tell, it's not a completely solved one. Bayesians usually agree about their calculations, but there's heavy debate about the "philosophy".…

I was thinking it through more and I think it's the stopping procedures that might not halt that are the problem. You can have a data dependent stopping procedure if it's guaranteed to halt which makes sure all data does get counted.

For example of one that might seem bad, but does halt, and turns out to be OK:

Flip a coin until you have more heads than tails OR reach 500 flips.

This procedure will produce a majority of trials with more heads than tails, but I think the average over many trials will be 50/50. The conceptual reason is that stopping early sometimes prevents just as many heads as tails that would have come up after stopping. I haven't formally proved this but I did a simulation with a million trials with that stopping procedure and got a ratio of 1.0004 heads per tails which seems fine (and after some reruns, I saw a result under 1, so that is possible). Code here:

http://pastebin.com/H42qHYbA

With a guaranteed halt, a sequence of 500 tails and 0 heads can be counted. With no guaranteed halt, it's impossible to count a tails heavy sequence, which is not OK because it's basically ignoring data people don't like.

Does that make sense? I think it may satisfy the stuff you/Bayesians/Eliezer are concerned with. It means it's OK to stop collecting data early if you want, but you do need some rules to make sure your all your results are reported with no selectivity there.

There's also a further issue that these kinds of stopping procedures are not a very good idea. The reason is that while they are OK with unlimited data, they can be misleading with small data sets. It's like the guy who bets a dollar, and if he loses he bets two dollars, and if he loses again he bets 4 dollars (repeated up to a maximum bet of 1024 dollars). His expectation value in the long run is not changed by his behavior but he does affect his short term odds: he's creating an above 50% chance of a small win and an under 50% chance of a larger loss. If you only do 10 trials of this betting system, they might all come out wins, and you've raised the odds of getting that result despite leaving the long term expectation value alone. Doing essentially the same thing with scientific data is unwise.

BTW/FYI I believe I have no objections to the Bayesian approach to probability but I do think the attempt to make it into an epistemology is mistaken (e.g. because it cannot address purely philosophical issues where there's no data to use, so it fails to solve the general problem in epistemology of how knowledge (of all types) is created.)

Re: How Khan Academy is using machine learning to assess student mastery

#44

Post by equark » Thu, Nov 03, 2011, 11:54 PM UTC

xenophanes wrote: post #34

equark wrote: post #33

Earlier quoted context omitted.

You're stuck in frequentist thinking. "Bias" is a property of repeated sampling -- the expectation over repeated samples. But we just have one! The relevant question is what is your best guess for p, the probability the coin will be heads. Under a uniform prior [0, 1] the posterior mean is the empirical mean. How you sample is of no consequence. The likelihood/posterior f(p|#heads, #tails) is p^(#heads)(1-p)^(#tails)…

Consider an infinite string of coin flips. Now consider a subset selected by a stopping rule to meet a particular criterion. And a different subset chosen with an N=100 criterion. The first stopping rule creates a bias: you have a non-random sample chosen to meet that criterion. The second stopping rule doesn't do that, it gets what we call a "random sample". If someone then takes your dataset and assumes it's a rand…

Yes, clearly. I stated that too strongly. Sampling procedures can definitely matter enormously, but stopping rules are within a class of ignorable rules. The link above gives a more precise definition.

Re: How Khan Academy is using machine learning to assess student mastery

#45

Post by equark » Thu, Nov 03, 2011, 11:58 PM UTC

ced wrote: post #42

equark wrote: post #33

Earlier quoted context omitted.

You're stuck in frequentist thinking. "Bias" is a property of repeated sampling -- the expectation over repeated samples. But we just have one! The relevant question is what is your best guess for p, the probability the coin will be heads. Under a uniform prior [0, 1] the posterior mean is the empirical mean. How you sample is of no consequence. The likelihood/posterior f(p|#heads, #tails) is p^(#heads)(1-p)^(#tails)…

Off-topic, but what happened to statsia? I was curious to see what you were working on.

Website is down but the project continues. Email beta@statsia.com if you want to be put on our insider's list.

Re: How Khan Academy is using machine learning to assess student mastery

#46

Post by gujk » Fri, Nov 04, 2011, 8:06 AM UTC

lliiffee wrote: post #40

eru wrote: post #39

Earlier quoted context omitted.

What did the earlier era use?

In a case like this, perhaps a perceptron?

Which is Ng's class's next chapter after logistic regression. :-)

Re: How Khan Academy is using machine learning to assess student mastery

#47

Post by gujk » Fri, Nov 04, 2011, 8:11 AM UTC

ahsanhilal wrote: post #24

My question is how does time dependency work in this case. I am trying to wrap my head around how a prediction engine would work when your assessing students on the basis of not just past/current performance but also how much time their taking while answering each question. I think you can model for randomness (kids getting lucky while answering a question), but if you can somehow add time-dependency to the model, th…

Time-to-answer is a numeric, but perhaps not linear, predictor.

Re: How Khan Academy is using machine learning to assess student mastery

#48

Post by xenophanes » Fri, Nov 04, 2011, 4:24 PM UTC

equark wrote: post #44

xenophanes wrote: post #34

Earlier quoted context omitted.

Consider an infinite string of coin flips. Now consider a subset selected by a stopping rule to meet a particular criterion. And a different subset chosen with an N=100 criterion. The first stopping rule creates a bias: you have a non-random sample chosen to meet that criterion. The second stopping rule doesn't do that, it gets what we call a "random sample". If someone then takes your dataset and assumes it's a rand…

Yes, clearly. I stated that too strongly. Sampling procedures can definitely matter enormously, but stopping rules are within a class of ignorable rules. The link above gives a more precise definition.

I think that you are mostly right about halting (guaranteed) stopping rules.

See my other comment, up a few times then down the other branch, the one with the pastebin code.

However the example with the two doctors was not the halting type.

Can you agree to that? Or do you have a defense of non-halting stopping rules, even though they are incapable of reporting some data sets?

I think I figured this out but would be interested in criticism on this point if not. Is there some way of dealing with non-halting that makes it OK?

The book says if there's a stopping rule then inferences must depend only on the resulting sample but that assumes there is a resulting sample -- that the procedure halts.