Live data from Hacker News

New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

intextbooks.science.uu.nl

101–110 of 126 posts

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#101
post #58

I am somewhat skeptical of this. First, the headline result of 0.7*sigma improvement is the output of a statistical based on lessons/reviews they engaged with and their mid-term score, with that shift being for "full engagement". Based on their tables something like ~16 students (11% of the group) actually reached that level of engagement Second, trying to incorporate past grades into their modelling is not a substit…

Worse, because students complained about the difficulty of the AI-graded quizzes, they switch to multiple-choice questions only, which increases engagement, but after analyzing the exam results they determine that multiple-choice questions don't seem to help and add AI-graded questions back, after which engagement drops again. That means their experiment design is partially caused by their results instead of the othe…

[dead]

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#102

Earlier quoted context omitted.

Lots of people in education will happily tell you how the past 15 years of tech integration has been a net negative. There ARE technologies that have improved things, but so much high-cost useless tech has been shoved into every level of education that many educators are incredibly leery of new tech. The issue is that while the underlying technology is useful, the way it gets integrated is frequently not. An administ…

I had a chance to use Google Classroom for a non-profit I was volunteering with, and wow it sucks. If that is what teachers and students have to contend with, yeah, I'd push back against any and all tech forced on me as well. It's all well intentioned, but the road to hell is paved with them.

This, ALL DAY LONG.

Having significant roles in the design, funding, and implementation of various public and private school educational projects which integrated some minimal technology I have seen this firsthand as far back as it has existed.

The schools which had the GOOGLE Classroom crap 'sold' to administrators and forced upon the teachers and students were examples of absolutely abysmal failure at every role to which they were supposed to help.

GC is s complete waste of the schools limited budgets, a waste of instructors already over-committed time, and a complete failure to deliver improved educational outcomes for the students. They may have been more 'engaged' in the screens, but they learned less. In the end the GC framework becomes a leash, used by schools to monitor and invade the out-of-school aspects of student lives, and tilts the field more steeply where disparities of home income levels exist.

My particular programs' inclusions of technology were never for 'education' with the exception of elective CS and robotics classes. Otherwise the technology was for records management and course content preparation help for teachers.

One project's tech use case stands out distinct from that limit. A charter school project - instructors and students working to create video and online course materials documenting the conversion of a school program from single grade standard 'science' classes to multi-age, collaborative, hands-on STEM science programs.

But the tech use and coursework were more for documentation and dissemination of the conversion process than for the principle educational objectives.

Actual students did use adapted legacy Samsung android devices to record observations and collaborate among peers during the observations of field studies and workshop classes.

The handhelds checked out to students' on a 'per project' basis each day, so there was no 'individual content' on any of the devices.

The repurposing of these devices wasn't accomplished through 'locking down' an overcapable platform, but by building from the kernel up and only including the specific functionality as required to collect and collaborate - so, no web, no chat, no games, no distractions. The specific workshop content and objectives were available via the handheld, and at specific places in the progression the students were required to input observations and record still or videos. Those images were not shared among students and there was no incentive for shenanigans as all content was directly attached each student's workshop performance and we were clear in the start of the classes that because this was going to be part of the conversion process documentary, no people in any film. we has a big library of already categorized required captures: Plants in various stages, equipment set-ups, insects in situ for ID, test-strip color comparisons, reagent container labels, etc. Some of the kids called them something from a scifi TV show, complete with "Beam me outta here" jokes.

So, technology can be very useful, but not when it comes with external motives and further widens socio-economic disparity. If the objective is MORE engaging content and MORE accurate records of student progress, then sure.

But, that usefulness is usually directly inverse to the cost and functionality of the devices: cheaper, limited capability tech will usually if not always outperform fancier, overcapable tech.

A $very-cheap refurbished Samsung Galaxy stripped down to whitelisted bluetooth and Wifi LAN, with an distributed storage layer and local-only browsing (Apache Cordova based app) can get alot done without adding tools not required for the specific needs of the student [or educator in many cases].

Campus server infrastructure de-minimus was a solved problem with bittorrent sync and each device only subscribing to the specific file-tree required for the specific workshop/class, and student submissions are only accessible to that student, and the course instructor(s) - eliminates the bullying and other misuses of technology.

We also were able to use a direct approach to resolve the personal phones in class problem, at least for the part f the campus participating in the STEM conversion (4th to 7th grades). Students had to turn off phones, lock them in a pouch, and hand that in to receive a handheld.

If a parent needed to reach the student in school, that's why the school front office had phones. Otherwise, student's were 'in class' and not to be disturbed.

But this was over ten years ago, and today's parents and students would probably throw a toddler sized tantrum at these rules.

Too bad the project got rug-pulled for funding in between years 1 and 2 for the documentation and distributable methods layer; but the school did get to keep teaching the hands-on STEM focus to completion of the then enrolled students - we baked in the costs for that into the first year at the insistence of the participating teachers and parents.

TLDR; don't let your school get SCROOGLED.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#103
post #58

I am somewhat skeptical of this. First, the headline result of 0.7*sigma improvement is the output of a statistical based on lessons/reviews they engaged with and their mid-term score, with that shift being for "full engagement". Based on their tables something like ~16 students (11% of the group) actually reached that level of engagement Second, trying to incorporate past grades into their modelling is not a substit…

Worse, because students complained about the difficulty of the AI-graded quizzes, they switch to multiple-choice questions only, which increases engagement, but after analyzing the exam results they determine that multiple-choice questions don't seem to help and add AI-graded questions back, after which engagement drops again. That means their experiment design is partially caused by their results instead of the othe…

I agree with all the criticism, but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students. It might even be considered unethical to withhold a tool that would be already available just to do research, once there is at least some evidence or very strong suspicion that it might be, in fact, beneficial for at least some of the learners (i.e., you cannot forbid students to read books from the library just because you want to do research). Using engagement instead of results is a common choice of proxy to acquire more data, but it can become a vanity metric quickly.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#104

Earlier quoted context omitted.

It feels to me that the venn diagram between "students that fully engaged with the material" and "students that learned well from the material" is going to basically be a circle for any teaching method.

The tricky question at the forefront of education research is, at least in my mind, trying to thread the needle between effective techniques that students don’t like, ineffective techniques that students do like, and poorly defined techniques that administrators like. And on top of all that student self-reports aren’t actually very reliable indicators of learning progress at all!

>effective techniques that students don’t like

Do students not like Mastership techniques?

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#105

Earlier quoted context omitted.

Today I saw a demo of Remarkable turned into Voldemort's diary from Harry Potter - you write to it, and it writes back, in handwriting.

This has been around for a bit, not sure it writes back in handwriting. https://github.com/awwaiid/ghostwriter Installation not for the average user.

Here's the new demo and repo: https://github.com/MaximeRivest/Riddle It writes back in a cursive font

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#106

I currently study Multivariate Calculus by using very new and nodern method: I read the text book, while solving the examples of the general for,ulas, or try to come up with my own. Then I do a s$ht-ton of exercises. I only use LLM’s to quickly clarify confusing topics or notation, but not really much else. I cancelled my Claude subscription. Now I use just Mistral and local Vibethinker-3B, but they work just fine. E…

In the Matrix movies you can upload complex knowledge to a person in seconds. If LLMs could help with this even a little it would be useful. But alas, people like myself who are slow learners still get overwhelmed when trying to learn too much or too fast without rest breaks.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#107
post #103
post #58

Earlier quoted context omitted.

Worse, because students complained about the difficulty of the AI-graded quizzes, they switch to multiple-choice questions only, which increases engagement, but after analyzing the exam results they determine that multiple-choice questions don't seem to help and add AI-graded questions back, after which engagement drops again. That means their experiment design is partially caused by their results instead of the othe…

I agree with all the criticism, but I'd like to point out that this kind of study must be done in a way that doesn't discriminate any of the students. It might even be considered unethical to withhold a tool that would be already available just to do research, once there is at least some evidence or very strong suspicion that it might be, in fact, beneficial for at least some of the learners (i.e., you cannot forbid…

It's indeed difficult to do good education research, and instructors prefer to just mess around without a control group instead. But the intellectually humble thing to do would be to admit that the experiment can't differentiate between a method that turns bad students into good ones and one that merely identifies good ones through selection effects. When more conscientious researchers get around to doing a proper randomized controlled trial on an intent-to-treat basis, it invariably turns out to have been the latter.

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#109
post #29

Yes! Very exciting to see this. Bloom's Two Sigma Opportunity suggests that there's another SD improvement available: https://en.wikipedia.org/wiki/Bloom%27s_2_sigma_problem

As far as I know, replications showed an effect size that's still pretty amazing by educational standards but nowhere near z=2.0 (I think z=0.6 is the current best guess).

Re: New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

#110
There's a famous post by Erik Hoel that calls the human version of this Aristocratic Tutoring [1] (Scott Alexander is unconvinced [2]).

In the 1980s, a researcher called Benjamin Bloom claimed a z=2.0 (that is 2σ) advantage for a combination of mastery learning (don't move on the the next topic until you've mastered the current one) and 1-on-1 tutoring. Later replications show there is definitely something going on, but the effect size is much lower, for example around z=0.7 in a 2020 paper [3].

I'm still open on AI tutoring, though the Dartmouth results look impressive. Someone please try and replicate this.

There's a saying that AI helps the best students get better, and the worst ones get worse. (Anthropic sort-of agrees [4].) It'll be interesting to see how that turns out.

[1] https://www.theintrinsicperspective.com/p/why-we-stopped-mak... [2] https://www.astralcodexten.com/p/contra-hoel-on-aristocratic... [3] https://www.nber.org/papers/w27476 [4] https://www.anthropic.com/research/AI-assistance-coding-skil...

Post reply on HN