Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI boosted homework scores, then exam scores dropped: study

economist.com

Post reply on HN
Print view View original

241–250 of 409 posts

Re: AI boosted homework scores, then exam scores dropped: study

#241

Post by doctorpangloss »

kzz102 wrote: post #214
doctorpangloss wrote: post #197

Earlier quoted context omitted.

well most university exams are designed to measure how much you study. so we didn't really need a study to tell us, "Exams continue to measure what they are designed to measure." they're not designed to measure general aptitude, or function as admissions criteria, or screen for job applications, or any other numerous things they are used for. there can be many questions of pedagogy. one of them is, what do our exams…

The fact that exam scores are correlated with how much you study is not the same as exams only reflect how much you study. Two students who study the same amount could have very different exam scores. The reason that there still is a strong correlation between exam score and time of study is because if all other things being equal , students who studies more have higher exam scores.

i'm not saying they reflect how much you study. they reflect a lot of things, including that. but you ask the people who write the tests, they're going to say, how much you know or how much you study, but nonetheless, they are limited. i agree with you. that's part of my point.

let's imagine a different study. we instead compare AI-users and non-users on a Wechsler (IQ-adjacent) test.

overall, it would be surprising if AI usage impacted your Wechsler scores. someone has done this study and the impact is quite quite small. BUT. do we care? We don't use Wechsler scores for admissions, we don't use them for jobs, we don't use them for... are you getting it now? A Wechsler family test is measuring something real, just like a university exam measures something. But what do we USE them for? Wechsler and a typical university exam are, in some senses, EQUALLY vague in terms of their fitness for purpose for answering a question like, "should we hire this guy?"

Like there is an association between IQ and earnings but it is actually surprisingly small! There is an association with math education and earnings and it is also surprisingly small. And consider how many people get by just fine without using a single piece of math education once they have finished school - like what if maximizing your earnings isn't all that it is about? Are you getting it now?

The issue isn't the AI usage. I can find tests that are immune to AI usage. The issue is using tests for things that they are not designed for. We pick and choose, for some subtle but nonetheless pervasive cultural reasons, which tests we use for which purpose, and very frequently, not because they are calibrated for the chosen purpose. This is coming from someone who scores very well on all these tests, and have kids, so I have a very strong incentive to buy into the status quo, and I'm telling you: academic testing has been fucked up for a long, long time.

Re: AI boosted homework scores, then exam scores dropped: study

#242

Post by a2ff6eeb0 »

insane_dreamer wrote: post #235
a2ff6eeb0 wrote: post #14

[flagged]

We could also drop school altogether and just make sure every kid has an LLM-connected device that's voice driven. They wouldn't even need to learn to read and write -- that's old school. update: this would make universal education free! because MegaCorp would happily give/support the devices free of charge (in exchange for ads being inserted into the responses)

If we actually manage to create superintelligence that makes human mental labor unable to compete economically, can you give me a reason that humans should value education? I personally think there's some inherent fun in learning, but that's not a reason to impose my hobby on everyone.

I think we're working very hard to build a world where there's no real reason for education.

Re: AI boosted homework scores, then exam scores dropped: study

#243

Post by dan_ggggg »

shimman wrote: post #199
elictronic wrote: post #147

Earlier quoted context omitted.

You aren’t being cynical, you’re arguing with disingenuous entities with money on the line. We already know it’s used primarily for the negative case. Everyone who ever intended High School or College knows this.

Definitely had a sheltered public school experience, took AP stats my senior year and realized all the top students were sharing answers from an earlier period through text messages. After figuring this out I too joined in on the action, then shortly after I quickly deskilled. Suppose it's good to learn how elastic the brain is, in both directions, at a young age where it doesn't matter.

> at a young age where it doesn't matter.

Doesn't it matter the most at a young age?

Re: AI boosted homework scores, then exam scores dropped: study

#244

Post by idontwantthis »

baud9600 wrote: post #238

I am waiting for AI Viva Voce (“AVV”). I am really surprised we aren’t hearing more about this idea? Try it for yourself by making a prompt like this (adapt as needed): Pretend I am an undergraduate student of Computer Science. I am learning about early microprocessors from the 1970s. I want you to ask me an examination question as if you were doing a viva voce exam with me, to test my understanding of concepts. I wa…

I've tried stuff like this, but the problem is that you don't know if it's answers are correct, and more importantly, you don't know if its questions are on base or not. Plus its too easy to go off on tangents, especially when you ask it questions to clarify. You can get some more success by writing down a framework beforehand, but openended dialog is still not a good way to learn with AI, from my experience.

Re: AI boosted homework scores, then exam scores dropped: study

#245

Post by Avicebron »

baud9600 wrote: post #238

I am waiting for AI Viva Voce (“AVV”). I am really surprised we aren’t hearing more about this idea? Try it for yourself by making a prompt like this (adapt as needed): Pretend I am an undergraduate student of Computer Science. I am learning about early microprocessors from the 1970s. I want you to ask me an examination question as if you were doing a viva voce exam with me, to test my understanding of concepts. I wa…

I mean, everyone I know has been using socratic dialogue for months to aid understanding, this seems like a variation of that. The issue is that if the answer is too easy to obtain it's hard to have the discipline not to cheat.

Re: AI boosted homework scores, then exam scores dropped: study

#246

Post by tehnoslow »

Honestly I wouldn't demonize AI in education. When we were students we were forbidden from using calculators, then the internet and now it's become the norm. It's the same with AI, I think. AI has already become a part of our lives. The important thing is just teaching children how to use it correctly.

Re: AI boosted homework scores, then exam scores dropped: study

#247

Post by a2ff6eeb0 »

add-sub-mul-div wrote: post #17
a2ff6eeb0 wrote: post #14

[flagged]

Sure. Let's not think too hard about what it means that general thinking and creativity are the skills we've stopped practicing this time. It will be fine. (You may be making the same point as me or the opposite point, I can't tell what's sarcasm anymore. I try to make mine obvious.)

Humans aren't going to be doing a better job of thinking and creativity in the long run compared to AI; there's fun in doing it for yourself, but that's not a great reason to force it on everyone, rather than just those who find it fulfilling. There'll be plenty of people who still think and create as a hobby, but if the AI can cover the creativity that the economy needs, why not let it?

I'm pretty sure people will consume the creative output of AI, since it'll be better able to make things catered to precisely what they find interesting. Human creativity is on its way out.

Re: AI boosted homework scores, then exam scores dropped: study

#248

Post by dataflow »

heddhunter wrote: post #229
dataflow wrote: post #215

Earlier quoted context omitted.

> I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place? Citation needed? I have no clue where you got this from. I hadn't even heard of it as a conjecture, let alone as something anyone accepted, let alone as generally accepted...

https://www.tandfonline.com/doi/pdf/10.1080/00220973.2012.74...

Before I dive into the paper: the claim was that some effect was generally accepted, not that a study was performed on it and that it drew a particular conclusion. Is this actually going to establish the former or are we just assuming that the existence of a study implies general acceptance of its conclusion?

Re: AI boosted homework scores, then exam scores dropped: study

#249

Post by bonoboTP »

doctorpangloss wrote: post #241
kzz102 wrote: post #214

Earlier quoted context omitted.

The fact that exam scores are correlated with how much you study is not the same as exams only reflect how much you study. Two students who study the same amount could have very different exam scores. The reason that there still is a strong correlation between exam score and time of study is because if all other things being equal , students who studies more have higher exam scores.

i'm not saying they reflect how much you study. they reflect a lot of things, including that. but you ask the people who write the tests, they're going to say, how much you know or how much you study, but nonetheless, they are limited. i agree with you. that's part of my point. let's imagine a different study. we instead compare AI-users and non-users on a Wechsler (IQ-adjacent) test. overall, it would be surprising…

You're right that academic exams also measure something along the lines of instruction following / obedience / willingness to jump over hoops for no good reason etc. and that's often a good signal for most kinds of jobs.

Re: AI boosted homework scores, then exam scores dropped: study

#250

Post by mmargenot »

Glad to have scientific results on this, though in my view you could get this from first principles. Homework is onerous but it forces you to learn the material and get it into a configuration that works in your head, which you then validate with the exam.
Post reply on HN