Live data from Hacker News

AI boosted homework scores, then exam scores dropped: study

economist.com

151–160 of 409 posts

Re: AI boosted homework scores, then exam scores dropped: study

#151
post #96
post #86

Earlier quoted context omitted.

Okay, but this article isn't about that: > The study followed 27,000 pupils aged 12 to 18

The same applies to high school students. Just make it exam based and give suggested reasons. (Third grade is 9-10 years old I believe.)

Too many of them don't really care. Otherwise they'd be doing the homework for real before they cheat it.

Re: AI boosted homework scores, then exam scores dropped: study

#152

Earlier quoted context omitted.

This is my thoughts as well! It makes smarter people smarter and dumb people dumber!

It helps smart people be barely more effective and helps dumb (more importantly, people who do not value effort, people who are lazy, people who are self absorbed) people shit out endless streams of worthless tokens that can swamp out everything. Raising the noise floor like this only makes it that much harder to find "Smart" people, which we were already doing terrible at. I use Claude every single day, but this is…

All I've noticed is AI creating a perverse incentive to make everything as complicated and bureaucratic as possible, so only the people that know how to leverage AI to cut through it ever succeed.

I'm using Claude at work myself and am impressed with the product, but notice that this is the only reason I need to use it at all. Our product pages were shit to begin with, now they're AI-generated and somehow even worse. Our procedures are incomprehensible spaghetti with enough arbitrary context switching to give a sadistic Soviet municipal administrator an erection at the thought of watching anyone try to actually follow them.

Use AI to create inefficiencies, then use AI to bypass them. Those who can't do the latter will struggle to survive.

Re: AI boosted homework scores, then exam scores dropped: study

#153
post #10

Taking a forklift to the gym increases the number on the bar, but mysteriously your gains go down.

To extend this analogy:

Using the forklift as a spotter and to assist in loading weights increased gains. Then again, a human can do all those things, and provide real human connection.

Re: AI boosted homework scores, then exam scores dropped: study

#154

Earlier quoted context omitted.

This is my thoughts as well! It makes smarter people smarter and dumb people dumber!

It helps smart people be barely more effective and helps dumb (more importantly, people who do not value effort, people who are lazy, people who are self absorbed) people shit out endless streams of worthless tokens that can swamp out everything. Raising the noise floor like this only makes it that much harder to find "Smart" people, which we were already doing terrible at. I use Claude every single day, but this is…

> Raising the noise floor like this only makes it that much harder to find "Smart" people, which we were already doing terrible at.

Clarification: to value “smart” people, which we were already doing terrible at.

Re: AI boosted homework scores, then exam scores dropped: study

#155

Moneyshot: "The results are eye-opening. After six months, pupils using ai saw their average homework score rise by 18% across all subjects. The time they took to complete each assignment fell from an average of 64 minutes to 45. But come exam time, the same students scored 20% below their classmates who had not called on ai’s help. Homework scores once predicted exam performance; now those who score highest are, per…

> Homework scores once predicted exam performance As with most training the journey is the point, not the destination. That said, I think smart use of AI could help. It could explain concepts in a way that might help you understand better, it could probe your knowledge in a more dynamic way by tailoring questions, and so on. This requires the AI be restrained by some harness, not free to write down the answers for yo…

> This requires the AI be restrained by some harness

How would this magic harness look like and why would anyone use it?

Re: AI boosted homework scores, then exam scores dropped: study

#156
post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

This is my thoughts as well! It makes smarter people smarter and dumb people dumber!

There is zero evidence for this.

LLMs make significant mistakes frequently and smart people have no way of judging those mistakes outside their domain expertise. They are also sycophantic and great at being an echo chamber which makes people feel smart even if they are not.

So I think the burden of proof is on you to prove that they somehow amplify intelligence, it seems highly unlikely.

Re: AI boosted homework scores, then exam scores dropped: study

#157
post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

It's not so clear that AI is an amplifier.. The paper has some fascinating analysis on this topic:

"At the other end of the distribution, AI students who spend more than 65 minutes on their homework receive homework and exam scores similar to those of non-AI students, suggesting that these students do not use generative AI for homework assignments. However, this group consists entirely of students who adopted generative AI no more than Öve months. Six months after adoption, no AI student spends more than 65 minutes completing their homework (see Figure A5). This is consistent with the gradual process of learning how to use AI tools. It also suggests that AI crowds out the highest level of e§ort."

"Interestingly, in the range of 50-65 minutes, the median and the interquartile range of exam scores of AI and non-AI students are similar. This implies that, in the range where AI students and non-AI students have overlapping homework times, students who spend the same amount of time completing homework on average receive similar exam scores."

"This pattern shows that students who spend the same amount of time on homework learn similarly, with or without generative AI. In other words, generative AI reduces time spent learning for the majority of AI students but not learning efficiency for those who spend the same time studying as the non-AI students."

Re: AI boosted homework scores, then exam scores dropped: study

#158

Earlier quoted context omitted.

This is my thoughts as well! It makes smarter people smarter and dumb people dumber!

There is zero evidence for this. LLMs make significant mistakes frequently and smart people have no way of judging those mistakes outside their domain expertise. They are also sycophantic and great at being an echo chamber which makes people feel smart even if they are not. So I think the burden of proof is on you to prove that they somehow amplify intelligence, it seems highly unlikely.

Most domains have some kind of internal consistency/theory building you can do. A smart person can certainly notice inconsistencies when trying to learn something. In fact they're likely to be points of confusion that the smart person will dive into just to try to make sense of things, even if they don't suspect the LLM is at fault.

Re: AI boosted homework scores, then exam scores dropped: study

#159

Guess what this is probably fine. AI helps people do their work on a daily basis and probably makes them memorize less things. That’s how the world is moving. You don’t need to be able to memorize something for an exam to show mastery of a topic. And we shouldn’t organize our society around memorizing things for exams.

Mastery requires memorization.

You can't play an instrument without muscle memory.

Why are commercial airline pilots required to spend hundreds of hours in simulators? So that when things go wrong they don't have time to think, they just react. And their reaction will be the correct course of action because that's what they have drilled on.

How many times have you gone "Oh, this new thing I have just encountered reminds me of this other thing?". How can you make such leaps without knowing that other thing?

Re: AI boosted homework scores, then exam scores dropped: study

#160
post #24

Earlier quoted context omitted.

Meetings are still like exams, if you can't craft a line of thinking on the spot it doesn't come off well. You don't need a final answer but you can't just answer "I don't know" otherwise you risk being slimed (shout out to anyone who still remembers "You Can't Do That on Television")

“I don’t know but after this meeting I will find out and get back to you” Is an acceptable answer.

Not really, in any company I've ever worked.

The answer is "accepted", in the sense that it won't get you fired. But the meeting plows on, and decisions are made in absence of the answer to the question! — and never revisited once the answer is known.

And like 90% of the time it comes up, it's because the data contradicts the decision.

Post reply on HN