New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf] • Hacker News Board

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#121

Post by Herring » Mon, Jul 06, 2026, 7:26 PM UTC

guai888 wrote: post #87

Herring wrote: post #82

Earlier quoted context omitted.

Hmm, it might be better to just go straight to what leading countries currently do (Finland/Estonia/Japan/Singapore/etc). Other factors probably play a bigger role, like school funding, school autonomy, teacher professionalism (high education level + continuing education + good pay), free school meals/healthcare/transportation etc

One major limitation is the cost. It costs lots of money to train teachers and pay them good salaries. What leading countries are doing currently might not be sustainable for the long run. Cognitive abilities also different greatly from people to people in different discipline. There is no silver bullet.

America is like a trust fund baby given all the advantages and then the baby goes "fuck it, life is too hard, I am just going to do coke and die early”.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#122

Post by protocolture » Mon, Jul 06, 2026, 11:35 PM UTC

yorwba wrote: post #111

protocolture wrote: post #104

Earlier quoted context omitted.

>effective techniques that students don’t like Do students not like Mastership techniques?

Students don't like being told they're bad at something. Students don't like being told to do the thing they're bad at. They especially don't like having to constantly repeat this in a loop until they've caught up with the rest of the class, while other students get to enjoy their free time. Also, mastery learning doesn't look all that effective on standardized tests: https://www.jstor.org/stable/1170613

How do you do Mastership in a class environment? Do they leave half the class waiting for everyone else to catch up?

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#123

Post by sebastiennight » Tue, Jul 07, 2026, 12:56 PM UTC

cgearhart wrote: post #116

sebastiennight wrote: post #64

Earlier quoted context omitted.

So there might be zero effect? If it's purely a correlation, then maybe those students would be more successful than average even without the study group. They're already the most motivated kids. Maybe they just do "motivated kid stuff" and would still outperform.

Yes, that’s what I think at this point. There is no effect of the study group except as a support group. (That’s all it was for me when I was a student and joined the self-organized study group.)

My self-organized study group was quite a mix of characters, and while I don't doubt that a couple of my friends's grades did benefit from having me there to help them out, I do definitely think that most of the stuff I remember learning there was outside the boundaries of what any of our parents would have approved :)

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#124

Post by nkmnz » Wed, Jul 08, 2026, 9:55 PM UTC

mapt wrote: post #115

nkmnz wrote: post #103

Earlier quoted context omitted.

I agree with all the criticism, but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students. It might even be considered unethical to withhold a tool that would be already available just to do research, once there is at least some evidence or very strong suspicion that it might be, in fact, beneficial for at least some of the learners (i.e., you cannot forbid…

> but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students. This is definitionally impossible. Testing whether the tool benefits the students that get it is an attempt to benefit certain students at the expense of others. Get over this hangup if you want to do this sort of research.

Feel free to create your own university where you can do this sort of research in a discriminatory way. Have fun with the litigation afterwards. Also, explain to the peer reviewers that they just need to get over the hangup, too, when you try to publish your research.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#125

Post by nkmnz » Wed, Jul 08, 2026, 9:57 PM UTC

yorwba wrote: post #107

nkmnz wrote: post #103

Earlier quoted context omitted.

I agree with all the criticism, but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students. It might even be considered unethical to withhold a tool that would be already available just to do research, once there is at least some evidence or very strong suspicion that it might be, in fact, beneficial for at least some of the learners (i.e., you cannot forbid…

It's indeed difficult to do good education research, and instructors prefer to just mess around without a control group instead. But the intellectually humble thing to do would be to admit that the experiment can't differentiate between a method that turns bad students into good ones and one that merely identifies good ones through selection effects. When more conscientious researchers get around to doing a proper ra…

> I agree with all the criticism...

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#126

Post by mapt » Fri, Jul 17, 2026, 12:06 PM UTC

nkmnz wrote: post #124

mapt wrote: post #115

Earlier quoted context omitted.

> but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students. This is definitionally impossible. Testing whether the tool benefits the students that get it is an attempt to benefit certain students at the expense of others. Get over this hangup if you want to do this sort of research.

Feel free to create your own university where you can do this sort of research in a discriminatory way. Have fun with the litigation afterwards. Also, explain to the peer reviewers that they just need to get over the hangup, too, when you try to publish your research.

"Feel free to create your own university where you have the funding for petri dishes. In the meantime, I'm going to publish without them, because I feel like this tissue culture I'm growing on my lunch is statistically significant."

Just because I am not personally entitled (by funding, by ethics, by career position) to collect the necessary data to demonstrate a hypothesis, does not make that hypothesis demonstrable with lesser data.