New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
intextbooks.science.uu.nl
New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
1–10 of 126 posts
Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
#2Do you have a larger study planned for the Fall? It definitely seems promising.
I'm curious how well you feel this worked because the subject was Statistics (objective grading) versus something more subjective like Civics or Literature.
PS - I'd say this qualifies for Show HN, too!
Do you
Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
#3Very nicely typeset.
Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
#4Too bad the educational use case doesn't make any money. Good LLMs are a game changer for people motivated to learn.
Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
#5Honestly whether or not this was effective seems less important to me than the adoption numbers.
Text book reading in this course was 10-15% at baseline ... but this AI thing got 90% voluntary usage ungraded.
Even if its worse per-hour than a textbook, you're now teaching 6x as many students _something_ instead of teaching a small minority everything.
So really it just becomes an optimization problem at that point because most students are at least in the funnel/in the running to learn something.
The paper kind of proves this itself ... they tweaked the quize formats mid-semester and where able to iterate which you can't do on a textbook that nobody opens in the first place
Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
#6This is exciting because the effect size is so large. But as the author's acknowledged, selection bias is nearly impossible to control for in this non-randomized study:
> and lacks randomized controls. Self-selection is the central threat: students who complete more quizzes may be more motivated or higher-performing generally
But this is still a strong result. I'm excited to see more in this space.
Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
#7Too bad the educational use case doesn't make any money. Good LLMs are a game changer for people motivated to learn.
I don't want to learn from hallucinations where it will change its answers based on me questioning their teachings. I use it for conversations in a language I'm learning, but I quickly learned that asking it grammar questions for example is not a wise decision.
Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
#8Interesting, congrats.
Are you planning on opening access to Phosphor?
Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
#9Too bad the educational use case doesn't make any money. Good LLMs are a game changer for people motivated to learn.
Wikipedia doesn’t make much money but is still helpful. LLMs don’t need to make a whole bunch of money to be helpful.
Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]
#10Do you have a larger study planned for the Fall? It definitely seems promising. I'm curious how well you feel this worked because the subject was Statistics (objective grading) versus something more subjective like Civics or Literature. PS - I'd say this qualifies for Show HN, too! Do you
They were using Sonnet 4.6 for some fre form responses so that could be applied to something subjective.