Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI boosted homework scores, then exam scores dropped: study

economist.com

Post reply on HN
Print view View original

251–260 of 409 posts

Re: AI boosted homework scores, then exam scores dropped: study

#251

Post by dan_ggggg »

ACow_Adonis wrote: post #227
grey-area wrote: post #156

Earlier quoted context omitted.

There is zero evidence for this. LLMs make significant mistakes frequently and smart people have no way of judging those mistakes outside their domain expertise. They are also sycophantic and great at being an echo chamber which makes people feel smart even if they are not. So I think the burden of proof is on you to prove that they somehow amplify intelligence, it seems highly unlikely.

Why would a smart person go to an LLM for an answer they cannot judge or test, be succeptible to flattery and sycophancy rather than picking up on the emotional manipulation and being suspicious/sceptical of the interaction, or looking for support from an echo chamber rather than a Socratic opponent? All of those sound like flaws and defects of dumb people?

> Why would a smart person go to an LLM for an answer they cannot judge or test, be succeptible to flattery and sycophancy rather than picking up on the emotional manipulation and being suspicious/sceptical of the interaction, or looking for support from an echo chamber rather than a Socratic opponent?

Great summary of the flaws with LLM "research"/"reasoning". It's always trying to con you, and I question the literacy and intelligence of the people who can't see this.

Re: AI boosted homework scores, then exam scores dropped: study

#252

Post by dan_ggggg »

doctorwho42 wrote: post #136
the_real_cher wrote: post #11

Earlier quoted context omitted.

This is my thoughts as well! It makes smarter people smarter and dumb people dumber!

All at the cost of ... {List of negatives regarding the construction and powering of AI data centers }

The social costs are worse.

Re: AI boosted homework scores, then exam scores dropped: study

#253

Post by insane_dreamer »

a2ff6eeb0 wrote: post #242
insane_dreamer wrote: post #235

Earlier quoted context omitted.

We could also drop school altogether and just make sure every kid has an LLM-connected device that's voice driven. They wouldn't even need to learn to read and write -- that's old school. update: this would make universal education free! because MegaCorp would happily give/support the devices free of charge (in exchange for ads being inserted into the responses)

If we actually manage to create superintelligence that makes human mental labor unable to compete economically, can you give me a reason that humans should value education? I personally think there's some inherent fun in learning, but that's not a reason to impose my hobby on everyone. I think we're working very hard to build a world where there's no real reason for education.

> I think we're working very hard to build a world where there's no real reason for education.

I don't know who you mean by "we", certainly not "me". But yes, the MegaCorps are indeed working hard on this, because they want to be the ones to provide the devices. And governments would be happy too so long as they're aligned with MegaCorp because it solves the "citizens' access to much information results in critical thinking that might be critical of us" problem -- a problem China solved with their GFW and censorship.

Re: AI boosted homework scores, then exam scores dropped: study

#254

Post by oblio »

jedberg wrote: post #187
oblio wrote: post #155

Earlier quoted context omitted.

> This requires the AI be restrained by some harness How would this magic harness look like and why would anyone use it?

"I need help with my homework, but I don't want answers, just guidance on how to get to the answer". Put that in your agents.md.

I'm way beyond homework, but I do appreciate the compliment on seeming young!

I meant: what's going to stop the average lazy student from ignoring whatever harness the university recommends and instead use an unrestricted LLM, thus learning nothing?

Re: AI boosted homework scores, then exam scores dropped: study

#255

Post by mmargenot »

mmargenot wrote: post #250

Glad to have scientific results on this, though in my view you could get this from first principles. Homework is onerous but it forces you to learn the material and get it into a configuration that works in your head, which you then validate with the exam.

as a fun aside here from the abstract it does come down to how you use it:

> AI users who maintain similar homework completion time as non-AI users experience small learning losses.

If the pure completion time of homework goes drops significantly with an LLM tool, I suspect these students are spending the extra time turning the material over in different ways to internalize, which is interesting.

Re: AI boosted homework scores, then exam scores dropped: study

#256

Post by ebiederm »

moffkalast wrote: post #196
nerevarthelame wrote: post #149

Earlier quoted context omitted.

The scary thing is that 81% of AI users in this study were determined to be "outsourcing" their homework to the LLMs - and the rate increases the more exposure they had to LLMs. The "slightly higher" performance is based on statistically insignificant samples (between 4 and 20 students, depending on the context, out of the total population of 26,000): https://bsky.app/profile/benjaminjriley.bsky.social/post/3mt...

I thought that it was generally accepted by now that homework in the volumes that it is being assigned in the modern day was not found to be beneficial in any significant way in the first place? Maybe once all students start outsourcing it to AI it might finally die like it deserves to. People these days grow up with almost no free time for themselves, it's all school, sports/extracurriculars or homework nonstop. All…

What I remember hearing is that exercises and practice are needed to transform words on the page to something you know and master.

What I remember being questioned is does it make sense to do those exercises as homework or would they be better in school.

Or on the flip side should school get out early like 11 or noon, like I think the german gymnasium does and have all the exercises as homework.

The US system where children get out of school at 15:30 and still have a bunch of homework seems a little lopsided someplace.

The 3 month break in-between school years is definitely questionable.

I think the answer is a complex one because it intersects with personality and neurodivergence.

Depending on who you are and your family situation any form of homework can be a real challenge. Not because of what you are studying but because of how difficult it is to sit down and do anything you aren't passionate about. Certainly anyone with an executive function disability will have that challenge.

Re: AI boosted homework scores, then exam scores dropped: study

#257

Post by jedberg »

oblio wrote: post #254
jedberg wrote: post #187

Earlier quoted context omitted.

"I need help with my homework, but I don't want answers, just guidance on how to get to the answer". Put that in your agents.md.

I'm way beyond homework, but I do appreciate the compliment on seeming young! I meant: what's going to stop the average lazy student from ignoring whatever harness the university recommends and instead use an unrestricted LLM, thus learning nothing?

> what's going to stop the average lazy student from ignoring whatever harness the university recommends and instead use an unrestricted LLM, thus learning nothing?

Nothing until they start failing exams and maybe seeing the error of their ways. My suggestions wasn't for you, it was for the smart student who wants to use AI to enhance their learning but not have it do all the work.

But cheating isn't new. People have been cheating in school for centuries. It's easier now, but the consequences were always the same. At some point the chickens come home to roost and you pay the piper.

Re: AI boosted homework scores, then exam scores dropped: study

#258

Post by a2ff6eeb0 »

insane_dreamer wrote: post #253
a2ff6eeb0 wrote: post #242

Earlier quoted context omitted.

If we actually manage to create superintelligence that makes human mental labor unable to compete economically, can you give me a reason that humans should value education? I personally think there's some inherent fun in learning, but that's not a reason to impose my hobby on everyone. I think we're working very hard to build a world where there's no real reason for education.

> I think we're working very hard to build a world where there's no real reason for education. I don't know who you mean by "we", certainly not "me". But yes, the MegaCorps are indeed working hard on this, because they want to be the ones to provide the devices. And governments would be happy too so long as they're aligned with MegaCorp because it solves the "citizens' access to much information results in critical t…

"we" as in this society.

Re: AI boosted homework scores, then exam scores dropped: study

#259

Post by bcrosby95 »

bonoboTP wrote: post #228
kzz102 wrote: post #221

Earlier quoted context omitted.

My apologies for coming off as over-enthusiastic, I am currently obsessed with this study. Here is another quote: "The negative learning effects are larger for students with higher initial achievement. The differences in the estimated full (6-10 month average) effects are substantial, with a 50% gap between the most negative effect (-24 percent) for the highest tercile and the least negative (-16 percent) for the low…

Interesting. Top academic performance is mostly correlated with conscientiousness (willingness to work hard and keeping track of things) and intelligence. And I'd add motivation and interest to that too. I think if you take a physics class where the student is intelligent and intrinsically motivated through their own interest (I admit this is rare) then AI probably helps.

I think if they are intelligent, intrinsically motivated, and willing to pursue knowledge beyond what the class requires, it probably helps.

That last part is key. Intrinsic motivation doesn't mean you pursue it outside normal bounds. My kid loves soccer, its her second favorite thing in the world, she has an absurdly high tolerance for physical discomfort while playing, but she doesn't play it at home. There's other things she's rather do, such as play with her toys.

When you move the bar to something even less interesting to most kids like science, you're going to have a pretty huge falloff. You're basically selecting for kids who choose to do it in their spare time. I know a lot of smart kids (I run a boyscout troop, my wife a girlscout troop, both with lots of high achievers), and none of them do this.

Re: AI boosted homework scores, then exam scores dropped: study

#260

Post by wredcoll »

doctorwho42 wrote: post #136
the_real_cher wrote: post #11

Earlier quoted context omitted.

This is my thoughts as well! It makes smarter people smarter and dumb people dumber!

All at the cost of ... {List of negatives regarding the construction and powering of AI data centers }

If you added up the negative effect of every single "ai data center" you'd end up with less than a single coal mine.

It's good to think about second order impacts but this is not useful.

Post reply on HN