Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI boosted homework scores, then exam scores dropped: study

economist.com

Post reply on HN
Print view View original

191–200 of 409 posts

Re: AI boosted homework scores, then exam scores dropped: study

#191

Post by eastbound »

WalterBright wrote: post #177
tancop wrote: post #163

All AI did here is exposed an old problem in education. Kids are expected to get everything perfectly for the first time but a lot of them don't so the class gets easier next year to keep up pass rates. We need to restructure the system to treat failure as a signal instead of a disaster. Grades should come from hard randomized exams with unlimited retakes so one bad day won't hurt you. Homework should be optional mat…

> Homework should be optional material for self study At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams. But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I k…

> 1. never miss a lecture, no matter what

As a student of a top-tier French Master's degree, I consulted with a teacher to deal with exhaustion and to ask for a class rescheduling for my case. Explaining my situation, the teacher looks at me and interjected:

—Waitwaitwait. You... you went to all lectures!?

And he was right. Rather than following the curriculum, I should have developed my taste for various engineering topics and only used the classes as entertainment.

Re: AI boosted homework scores, then exam scores dropped: study

#192

Post by doctorpangloss »

WalterBright wrote: post #177
tancop wrote: post #163

All AI did here is exposed an old problem in education. Kids are expected to get everything perfectly for the first time but a lot of them don't so the class gets easier next year to keep up pass rates. We need to restructure the system to treat failure as a signal instead of a disaster. Grades should come from hard randomized exams with unlimited retakes so one bad day won't hurt you. Homework should be optional mat…

> Homework should be optional material for self study At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams. But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I k…

do you have any evidence that CalTech's pedagogy is immune to today's threats and trends in education everywhere else? (no)

Re: AI boosted homework scores, then exam scores dropped: study

#193

Post by ezst »

beambot wrote: post #120
shuwix wrote: post #62

Education sustem is not about the actual knowledge, but about training various aspects of the brain during kids development. So even memorizing literature have benefits, even though it doesn't bring any career use. If you skip this, well your brain won't develop as much at period of life it's able to do so.

Maybe. As the nature of assistive tools changes, so does pedagogy. Calculators are the easy example since most of us are (still) familiar. But there are others that were just as pronounced -- e.g. log tables and slide rules were transformational and instrumental at one time, and you've (probably) never used them. If you had used slide rules to solve your homework problems then were excluded from using them on exams,…

> As the nature of assistive tools changes

I mean, does it? The way I see it, the best LLMs have to offer is infinite patience (until they inevitably and unpredictably start to confabulate, which should be a gigantic red flag, anyhow), I don't see how LLMs can be used to explore teaching paradigms that haven't been explored before. We know pretty well from centuries of empirical experimentation how children's brains develop under different stimuli. It's not exactly something the software industry needed to "hack".

Re: AI boosted homework scores, then exam scores dropped: study

#194

Post by nick486 »

mpalczewski wrote: post #185

When you get out into the real world, what will matter is how effectively you use AI. When you are done with school, you can take the "exam" using AI. The problem is that teaching is trying to teach skills that are no longer relevant and is always behind what the latest going on in the industry. Every time without fail you got interns, who after two years at stanford would learn more in the 3 months on a google inter…

Way before AI, the problem was very similar - universities do not teach practically useful stuff. Its been a permanent complaint about the education system, really, at least for the last 50 years.

AI just gave the students a way to dodge the slog.

But university was never intended to teach the bleeding edge. That would really be impossible in practice. Pre-phd, it is supposed to teach ways to efficiently attack a problem. you can take a bunch of "play courses", and succeeding at any of those requires pretty much only that one skill.

Nowadays, the challenge they face, is to keep teaching "problem attack methods" in a way that can't be trivialized by AI.

Though fundamentally, if you go to university, and evade learning the one thing you can learn there - thats your loss.

Re: AI boosted homework scores, then exam scores dropped: study

#195

Post by ezst »

closetheloopdev wrote: post #134

Students seem to optimize for local maxima/short-term rewards (homework scores) at the expense of the global maximum/long-term rewards (learning).

Which was pretty much the whole damn premise of needing a teacher to begin with. What a surprise!

Re: AI boosted homework scores, then exam scores dropped: study

#196

Post by moffkalast »

nerevarthelame wrote: post #149
GPerson wrote: post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

The scary thing is that 81% of AI users in this study were determined to be "outsourcing" their homework to the LLMs - and the rate increases the more exposure they had to LLMs. The "slightly higher" performance is based on statistically insignificant samples (between 4 and 20 students, depending on the context, out of the total population of 26,000): https://bsky.app/profile/benjaminjriley.bsky.social/post/3mt...

I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place? Maybe once all students start outsourcing it to AI it might finally die like it deserves to. People these days grow up with almost no free time for themselves, it's all school, sports/extracurriculars or homework nonstop. All worker drone and no play makes for an increasingly dysfunctional society.

Re: AI boosted homework scores, then exam scores dropped: study

#197

Post by doctorpangloss »

kzz102 wrote: post #157
GPerson wrote: post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic: "At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generat…

well most university exams are designed to measure how much you study. so we didn't really need a study to tell us, "Exams continue to measure what they are designed to measure."

they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for.

there can be many questions of pedagogy. one of them is, what do our exams measure and how do we use them? professors who say, "My exam is designed to measure who studies, not be used for all these other purposes that they are actually used for" - I don't buy it. It's the same as late night comedians saying they are not responsible for solutions, even when spending 90% of their air time making political jokes.

THIS is the pedagogical issue, that pedagogy has NEVER caught up with the scope of responsibilities. This is acute in STEM - I mean, the humanities departments are generally pretty well run, all things considered, in this regard. Generative AI is accelerating that pre-existing crisis.

Re: AI boosted homework scores, then exam scores dropped: study

#199

Post by shimman »

elictronic wrote: post #147
GPerson wrote: post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

You aren’t being cynical, you’re arguing with disingenuous entities with money on the line. We already know it’s used primarily for the negative case. Everyone who ever intended High School or College knows this.

Definitely had a sheltered public school experience, took AP stats my senior year and realized all the top students were sharing answers from an earlier period through text messages. After figuring this out I too joined in on the action, then shortly after I quickly deskilled.

Suppose it's good to learn how elastic the brain is, in both directions, at a young age where it doesn't matter.

Re: AI boosted homework scores, then exam scores dropped: study

#200

Post by WalterBright »

doctorpangloss wrote: post #192
WalterBright wrote: post #177

Earlier quoted context omitted.

> Homework should be optional material for self study At Caltech, homework was assigned but had no bearing on your grade. The grades were based on the midterm and final exams. But not mastering the homework usually resulted in flunking the exams. There were "retch" sessions after each homework assignment that was staffed by a grad student, and the purpose was to help the students understand the homework problems. I k…

do you have any evidence that CalTech's pedagogy is immune to today's threats and trends in education everywhere else? (no)

Nope.

When I was there, a long time ago, exams were timed and were usually open book open note. Blue books were filled in. You were trusted to adhere by those rules, and most students did their exams in their dorm rooms.

The evidence that the students honored the rules was some exams resulted in a 50% failure rate.

As for me, I went there because I wanted to learn the material. I did not care about getting a diploma. (Mine is in the basement somewhere.) I did not take any "easy A" classes, because I wanted a return on my time and tuition investment. (Though, easy A classes were hard to find at Caltech.) I wasn't even going to attend graduation, but my parents showed up and I attended to please them.

The classes, year by year, were dependent on mastering the previous year's classes. So if you cheat with AI, you're digging yourself into a bigger and bigger hole. Caltech rewires your brain. If you don't learn the stuff, you're going to be one of those EEs who carries around a card with V=A*R, V/A=R, V/R=A printed on it.

Post reply on HN