Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI boosted homework scores, then exam scores dropped: study

economist.com

Post reply on HN
Print view View original

281–290 of 409 posts

Re: AI boosted homework scores, then exam scores dropped: study

#281

Post by nullsanity »

asdfman123 wrote: post #279
kzz102 wrote: post #157

Earlier quoted context omitted.

It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic: "At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generat…

Personally in my own work it's pretty obvious how it's an amplifier. You waste far less time trying to find an answer to something that confuses you. You might say "it's good to learn research skills" and that's true to an extent, but tutors have always made people better students. And AI is a tutor you can message at any time, day or night, for free.

[dead]

Re: AI boosted homework scores, then exam scores dropped: study

#282

Post by [deleted] »

morpheos137 wrote: post #274
doctorpangloss wrote: post #197

Earlier quoted context omitted.

well most university exams are designed to measure how much you study. so we didn't really need a study to tell us, "Exams continue to measure what they are designed to measure." they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for. there can be many questions of pedagogy. one of them is, what do our exams…

I never studied in university and yet I acheived good exam scores. If you understand a topic and ave a reasonable memory and ability to apply my our understanding not much studying is required in my experience. If you don't understand the big picture then you got to laboriously keep track of and manipulate a bunch of disparate pieces.

[deleted]

Re: AI boosted homework scores, then exam scores dropped: study

#284

Post by Barrin92 »

asdfman123 wrote: post #279
kzz102 wrote: post #157

Earlier quoted context omitted.

It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic: "At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generat…

Personally in my own work it's pretty obvious how it's an amplifier. You waste far less time trying to find an answer to something that confuses you. You might say "it's good to learn research skills" and that's true to an extent, but tutors have always made people better students. And AI is a tutor you can message at any time, day or night, for free.

the difference in the vast majority of cases is that people go to a tutor with an intentional stance of cultivating a particular kind of practice, not merely to get an answer to a question. It's not technically impossible to do this with a chatbot but far less likely given that, unlike an even half decent tutor, a chatbot will never cultivate that attitude in you. The elimination of that friction is exactly why people talk to bots.

The good thing about a tutor is that he or she isn't always available and knows they won't be in the future, so they instill good habits and independence in you.

Re: AI boosted homework scores, then exam scores dropped: study

#285

Post by baud9600 »

idontwantthis wrote: post #244
baud9600 wrote: post #238

I am waiting for AI Viva Voce (“AVV”). I am really surprised we aren’t hearing more about this idea? Try it for yourself by making a prompt like this (adapt as needed): Pretend I am an undergraduate student of Computer Science. I am learning about early microprocessors from the 1970s. I want you to ask me an examination question as if you were doing a viva voce exam with me, to test my understanding of concepts. I wa…

I've tried stuff like this, but the problem is that you don't know if it's answers are correct, and more importantly, you don't know if its questions are on base or not. Plus its too easy to go off on tangents, especially when you ask it questions to clarify. You can get some more success by writing down a framework beforehand, but openended dialog is still not a good way to learn with AI, from my experience.

Good points. I’ve tried this a few times and I’ve been impressed by the dialogue. Given this was a very simple prompt, I am convinced a more precise version could be developed, including a rubric, a library of course content, starter questions and other guidance to the AI. I haven’t seen anything like this and schools / universities seem to be “struggling” with AI, which has me wondering why not embrace LLMs for learning and assessment, like this?

Re: AI boosted homework scores, then exam scores dropped: study

#286

Post by jkhdigital »

doctorpangloss wrote: post #241
kzz102 wrote: post #214

Earlier quoted context omitted.

The fact that exam scores are correlated with how much you study is not the same as exams only reflect how much you study. Two students who study the same amount could have very different exam scores. The reason that there still is a strong correlation between exam score and time of study is because if all other things being equal , students who studies more have higher exam scores.

i'm not saying they reflect how much you study. they reflect a lot of things, including that. but you ask the people who write the tests, they're going to say, how much you know or how much you study, but nonetheless, they are limited. i agree with you. that's part of my point. let's imagine a different study. we instead compare AI-users and non-users on a Wechsler (IQ-adjacent) test. overall, it would be surprising…

I just started teaching undergrad CS courses after ~20 years of various non-academic jobs and it took me about two semesters to realize that almost nothing matters except how I assess students.

Two weeks of ADHD-fueled research later, I concluded that academia is actively resistant to implementing assessment reform because it would expose the utter pointlessness of most of what happens in university classrooms.

The reality is that we have no idea what most university exams measure because they are ad hoc, written by amateurs (yes, most professors are untrained in pedagogical methods) with zero psychometric validity analysis.

Re: AI boosted homework scores, then exam scores dropped: study

#287

Post by CivBase »

tancop wrote: post #163

All AI did here is exposed an old problem in education. Kids are expected to get everything perfectly for the first time but a lot of them don't so the class gets easier next year to keep up pass rates. We need to restructure the system to treat failure as a signal instead of a disaster. Grades should come from hard randomized exams with unlimited retakes so one bad day won't hurt you. Homework should be optional mat…

You put way too much faith in kids' desire to learn and their ability to make responsible choices.

> Homework should be optional material for self study, evaluated by teachers if you choose to do it but never forced.

My wife is a teacher and her school does this. The result is that nobody does homework. The kids who need the extra practice don't get it and they fall into a spiral of failure and apathy. This is especially bad for subjects like math that build on themselves.

IMO the better solution is to require and grade shorter homework and provide students with optional, supplementary assignments that can be used to make up for missed credit on homework for the same material. That way students who grasp the material quickly can demonstrate and move on while others who need practice are naturally encouraged to get it without being permanently punished for struggling initially.

Re: AI boosted homework scores, then exam scores dropped: study

#288

Post by zemo »

Der_Einzige wrote: post #101
zemo wrote: post #18

abolish homework

The often underworked (most are autograding these days and reading old powerpoints rather than do real curriculum development) and (for the quality) overpaid teachers/professors will mass riot against this because it means they might have to actually teach. We might even get yet more anti-boy treatment than already exists among them as a social class in retaliation. Please also abolish PowerPoint and force oral exams…

I like school and think teachers are good and that your comment is the nonsensical ranting of a lunatic and as a result don't feel compelled in the slightest to read whatever it is that you linked.

Re: AI boosted homework scores, then exam scores dropped: study

#289

Post by embedding-shape »

Barrin92 wrote: post #284
asdfman123 wrote: post #279

Earlier quoted context omitted.

Personally in my own work it's pretty obvious how it's an amplifier. You waste far less time trying to find an answer to something that confuses you. You might say "it's good to learn research skills" and that's true to an extent, but tutors have always made people better students. And AI is a tutor you can message at any time, day or night, for free.

the difference in the vast majority of cases is that people go to a tutor with an intentional stance of cultivating a particular kind of practice, not merely to get an answer to a question. It's not technically impossible to do this with a chatbot but far less likely given that, unlike an even half decent tutor, a chatbot will never cultivate that attitude in you. The elimination of that friction is exactly why peopl…

> a chatbot will never cultivate that attitude in you

Hence the whole "AI is basically an amplifier of bad and good" argument made earlier. The ones who do this because they want to understand, don't need to cultivate this at all, it naturally happens with chatbots. They don't just ask for the answer to a question, but then dig into why it's like that and what not. But the ones that don't care, now have to do even less to get the fast answer without understanding.

Re: AI boosted homework scores, then exam scores dropped: study

#290

Post by jkhdigital »

moffkalast wrote: post #196
nerevarthelame wrote: post #149

Earlier quoted context omitted.

The scary thing is that 81% of AI users in this study were determined to be "outsourcing" their homework to the LLMs - and the rate increases the more exposure they had to LLMs. The "slightly higher" performance is based on statistically insignificant samples (between 4 and 20 students, depending on the context, out of the total population of 26,000): https://bsky.app/profile/benjaminjriley.bsky.social/post/3mt...

I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place? Maybe once all students start outsourcing it to AI it might finally die like it deserves to. People these days grow up with almost no free time for themselves, it's all school, sports/extracurriculars or homework nonstop. All…

The problem was never the volume of homework, it is the targeting and specificity. Learning happens through repetitive deliberate practice, full stop. You can’t learn through osmosis; effort is required.

Industrialized mass education has always suffered from a unit economics problem: the labor required to assign individually-tailored problem sets and manually grade them in the volume needed for most students to actually learn the material is prohibitively expensive.

Post reply on HN