Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI boosted homework scores, then exam scores dropped: study

economist.com

Post reply on HN
Print view View original

311–320 of 409 posts

Re: AI boosted homework scores, then exam scores dropped: study

#311

Post by mlyle »

asdfman123 wrote: post #279
kzz102 wrote: post #157

Earlier quoted context omitted.

It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic: "At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generat…

Personally in my own work it's pretty obvious how it's an amplifier. You waste far less time trying to find an answer to something that confuses you. You might say "it's good to learn research skills" and that's true to an extent, but tutors have always made people better students. And AI is a tutor you can message at any time, day or night, for free.

Finding an answer isn't always an objective, though. I agree it often has been too much in baseline schooling.

First, there's a certain amount of baseline knowledge we'd like students to possess. Without a certain prerequisite amount of underlying information committed to memory, it gets far more difficult to achieve fluency in a topic.

But more than that, there's other skills we're trying to build: frustration tolerance, processing contradictory information, disciplined problem solving. You only really develop these skills through productive struggle. If you find a way to shortcut the productive struggle, students truggle.

> but tutors have always made people better students.

Sure. Bloom showed us that students taught with a combination of tutorial and mastery methods, one-on-one, outperform students in a normal classroom by roughly 2 sigma.

The paradox has always been-- why hasn't technology unlocked these gains for students in normal classrooms? If we could boost everyone's performance by this amount, it would be huge for society-- but society can't afford to teach everyone with tutorial methods.

Since the 1970s, we've invested in edtech towards trying to make this happen, but most of it has actually had net-negative effects as best as we can measure. AI, so far, looks to be much worse.

I think part of the answer is that a big part of what makes a conventional classroom work are social pressures. So far, it looks like AI (and edtech in general) does more to dismantle conventional pedagogy and to break down the social fabric of the classroom, than it has improved differentiation or unlocked this tutorial effect more broadly.

Re: AI boosted homework scores, then exam scores dropped: study

#312

Post by WalterBright »

CobaltFire wrote: post #216
WalterBright wrote: post #177

Earlier quoted context omitted.

> Homework should be optional material for self study At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams. But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I k…

Naval Nuclear Power School had (maybe still does) this model 20 years ago as well. Pass rate for my class cohort was 28% over the entire 2 year pipeline. We didn't call them retch sessions, and they were taught by the instructors though. We also were encouraged to peer tutor and since we were all restricted to one building the homework was always group work allowed. Also had badges to track time spent in the building…

I think only a third or so of my class wound up graduating. Caltech doesn't flunk people out. What happens is students will transfer to easier schools, or drop out, or decide driving a truck is more to their liking.

Re: AI boosted homework scores, then exam scores dropped: study

#313

Post by kzz102 »

doctorpangloss wrote: post #241
kzz102 wrote: post #214

Earlier quoted context omitted.

The fact that exam scores are correlated with how much you study is not the same as exams only reflect how much you study. Two students who study the same amount could have very different exam scores. The reason that there still is a strong correlation between exam score and time of study is because if all other things being equal , students who studies more have higher exam scores.

i'm not saying they reflect how much you study. they reflect a lot of things, including that. but you ask the people who write the tests, they're going to say, how much you know or how much you study, but nonetheless, they are limited. i agree with you. that's part of my point. let's imagine a different study. we instead compare AI-users and non-users on a Wechsler (IQ-adjacent) test. overall, it would be surprising…

Education is a complex topic. I don't claim to understand it, nor do I think we can sort this out in a hacker news thread. What I reject is the simplistic claims like "exams don't mean anything". There are certainly exams that are badly designed, but there are also exams that are well designed. A well designed exam can look very badly to different people, based on what they know and where they come from. It's a bad idea to think exam scores as the single metric of education quality, it is equally bad to reject them completely, because chances are any alternative measurement people come up with are going to be worse.

Re: AI boosted homework scores, then exam scores dropped: study

#315

Post by Der_Einzige »

zemo wrote: post #288
Der_Einzige wrote: post #101

Earlier quoted context omitted.

The often underworked (most are autograding these days and reading old powerpoints rather than do real curriculum development) and (for the quality) overpaid teachers/professors will mass riot against this because it means they might have to actually teach. We might even get yet more anti-boy treatment than already exists among them as a social class in retaliation. Please also abolish PowerPoint and force oral exams…

I like school and think teachers are good and that your comment is the nonsensical ranting of a lunatic and as a result don't feel compelled in the slightest to read whatever it is that you linked.

What a great outcome of the school system!

Re: AI boosted homework scores, then exam scores dropped: study

#316

Post by jonahx »

sebastiennight wrote: post #269
jonahx wrote: post #225

Earlier quoted context omitted.

Depends who's using it. Like many tools the force multiplier depends on the operator. I'm confident it's an amplifier for people who know how learning works and already do a lot of it, successfully. However the level of "learning fluency" I'm talking about isn't reached for many until late college or grad school, and sometimes not at all. So I'm not surprised by the quoted results for 12-18 year olds.

> Depends who's using it. Like many tools the force multiplier depends on the operator. How would you prove/disprove this assumption without falling into a True Scotsman fallacy?

Fair question. In theory you could do an experiment where the subjects were grad students or professors, or top performing college students. Give them some fixed amount of time to understand some new subject on which they'll be tested, only 1 group has access to an LLM with appropriate context for the learning task, etc. That's just one idea...

Re: AI boosted homework scores, then exam scores dropped: study

#317

Post by Hnrobert42 »

It's an ad hominem attack, I know, but why am I not surprised that the Economist loves this?

True or not, it supports a narrative of "kids these days...", of a morally bankrupt educational system, of indulgent adults spoiling children, of just desserts to the wicked and lazy.

Again, I'm not saying the article is right or wrong. I'm saying I'm skeptical because it so tidily aligns with the Economist's editorial world view.

Re: AI boosted homework scores, then exam scores dropped: study

#318

Post by Izkata »

mlyle wrote: post #311
asdfman123 wrote: post #279

Earlier quoted context omitted.

Personally in my own work it's pretty obvious how it's an amplifier. You waste far less time trying to find an answer to something that confuses you. You might say "it's good to learn research skills" and that's true to an extent, but tutors have always made people better students. And AI is a tutor you can message at any time, day or night, for free.

Finding an answer isn't always an objective, though. I agree it often has been too much in baseline schooling. First, there's a certain amount of baseline knowledge we'd like students to possess. Without a certain prerequisite amount of underlying information committed to memory, it gets far more difficult to achieve fluency in a topic. But more than that, there's other skills we're trying to build: frustration toler…

> If you find a way to shortcut the productive struggle, students truggle.

I was assuming this was a typo and was thinking about making a joke about it, but it does appear to be slang that fits the context:

https://www.urbandictionary.com/define.php?term=Truggle

> 1. the standard of perpetual intellectual failure made by an individual.

> Truggle; the standard defenintion of a person who is a failureat everything.

Was it a typo or did you actually mean this?

Re: AI boosted homework scores, then exam scores dropped: study

#319

Post by intended »

GPerson wrote: post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

There’s been multiple articles at this point about AI’s impact on education.

It’s alarming how near universally AI has been a cataclysm in class rooms.

On homework the short answer is is that we give kids home work exercises.

AI helps learners reduce the effort expended to exercise and get results. This is making homework moot.

Re: AI boosted homework scores, then exam scores dropped: study

#320

Post by visarga »

Should have let the kids use AI during exam too, later in life they will be solving tasks with AI not manually. Just make the test questions difficult enough even with AI.

I see our current approach like a driving school training its students at fast running.

Post reply on HN