Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI boosted homework scores, then exam scores dropped: study

economist.com

Post reply on HN
Print view View original

371–380 of 409 posts

Re: AI boosted homework scores, then exam scores dropped: study

#371

Post by epgui »

GPerson wrote: post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

If you use AI to outsource your thinking, then AI is not “just an amplifier” at all.

A huge proportion of people who use AI use it to outsource their thinking.

Re: AI boosted homework scores, then exam scores dropped: study

#372

Post by GPerson »

epgui wrote: post #371
GPerson wrote: post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

If you use AI to outsource your thinking, then AI is not “just an amplifier” at all. A huge proportion of people who use AI use it to outsource their thinking.

That’s why I said it will be used more for bad than good. Maybe I should have said, “it will be used for far more bad than good for the purpose of human intellectual flourishing”.

It can be used for good for this purpose. You can use it to find primary source information. You can use it to generate challenges for you. You can use it to critique your work when there’s nobody else available. Etc etc.

Re: AI boosted homework scores, then exam scores dropped: study

#373

Post by ndriscoll »

grey-area wrote: post #369
ndriscoll wrote: post #355

Earlier quoted context omitted.

We're talking about learning here, i.e., building that knowledge and context. People also refer to more than one source, and I think you overstate the degree to which LLMs make mistakes in current year. Human teachers also make mistakes, and it's typical for a bright student who's paying attention to correct even small details.

LLMs are not a shortcut to mastery, in fact the opposite most of the time and their errors are qualitatively different from human errors.

I don't think anyone is claiming they're a shortcut to mastery. Personal tutors aren't a shortcut to mastery either. Both simply have the advantage of being able to deeply know where the student's knowledge frontier is, so they can tailor their answers to that, make connections and draw analogies to things the student has mastered (while pointing out distinctions), etc. The student still needs to put in the work.

Re: AI boosted homework scores, then exam scores dropped: study

#374

Post by conartist6 »

GPerson wrote: post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

Yeah the single thing it makes easiest is to think "i neither need nor want to do this because it is hard"

Evidently the AI users think hard things are beneath them. They want to win without doing any work. Sympathy, I have not.

Re: AI boosted homework scores, then exam scores dropped: study

#375

Post by Frieren »

josefritzishere wrote: post #94

I think we need to accept that AI causes cognitive decline, like Alzheimer's. For an Adult this may be manageable, but in a young brain, this is detrimental.

> I think we need to accept that AI causes cognitive decline

That is catastrophic because I heard several CEOs that said that we cannot do anything about it and that the technology is here to stay. So, I suppose that we will just need to adapt to a world with declining mental cognition for millions of children and a future with diminishing production capacity because ... checks notes ... this quarterly profits.

Re: AI boosted homework scores, then exam scores dropped: study

#376

Post by jonahx »

sebastiennight wrote: post #359
jonahx wrote: post #316

Earlier quoted context omitted.

Fair question. In theory you could do an experiment where the subjects were grad students or professors, or top performing college students. Give them some fixed amount of time to understand some new subject on which they'll be tested, only 1 group has access to an LLM with appropriate context for the learning task, etc. That's just one idea...

This experimental setup would only tell you whether this demographic (grad students or professors, or top performing college students) benefits from access to LLMs for acquiring a new knowledge or skill. It would not prove that "the results depend on the operator". To prove this (and figure out the traits that make a proficient operator) you would either need a massive dataset to which you'd apply some machine learni…

Yes, I'm using "high-performing college students" or "phd students" or whatever as a proxy for "people skilled at learning". It's not perfect but I think it'd be good enough. You could do lots of refinements on the idea. You could try to do some kind of "pre-test" that more directly tested "meta-learning skills".

My only point was I think it's possible to do. I personally think it's pretty clear. And there's nothing special about AI. In rural places, you could replace it with "access to books" or "access to a good tutor" and so on.

I also think you'd see the same effect with "motivation" (if you could test it), with performance on standardized tests, and with grades. AI will boost the better (by these metrics) students more.

Re: AI boosted homework scores, then exam scores dropped: study

#377

Post by freehorse »

GPerson wrote: post #7

The kids who used AI and studied for a similar amount of time as the non-AI high performers apparently had similar (slightly higher) performance. The kids who used AI and did poorly used the AI to do the homework for them (this is what the article says). I believe AI is basically an amplifier of bad and good. I’m cynical about the world and assume it will be used more for bad than good, but I don’t doubt some of the…

What AI definitely is as an amplifier (in my interpretation of the article) is an amplifier of "laziness": students spending less time studying, getting by with the homework with the less possible effort, and finally achieving worse results. And the longer one uses AI for, the "lazier" they get (the more they rely on it for homework, and study less).

If it is an amplifier of "good" vs "bad", I don't know, and I do not know what that means exactly. I do not think some students are just "bad" or "lazy" by character and that explains all. I think that their environment has a big role in shaping their actions. I don't think that AI cannot be used beneficially (and the articles talks about some attempts that maybe worked well), but it requires careful considerations, guidance and intentionality from educational institutions themselves. The same goes with all kinds of technology, and AI is nothing new in that sense, but often new technologies are introduced in education without careful considerations or evidence based policies. Which is why imo articles like this are very important.

Re: AI boosted homework scores, then exam scores dropped: study

#378

Post by Barbing »

j4yav wrote: post #326
Barbing wrote: post #190

Earlier quoted context omitted.

No evidence but somewhat of a counterpoints: Smart people know LLMs confabulate and tell them they’re Absolutely Right! Smart people don’t want to be embarrassed by trusting the hallucination machine and revealing their gullibility to others.

Smart people are handsome, charming, and always make the right decisions even on things they don’t know about. And if they ever are deceived, well I guess they were no true smart person.

Really widely reported in normie news and on reddit, hallucinations and sycophancy. Stand by that whether you’re a petroleum engineer or a bartender, if you’re clever you were probably either well informed or curious enough not to blindly trust the plagiarism machines.

Re: AI boosted homework scores, then exam scores dropped: study

#379

Post by fn-mote »

jkhdigital wrote: post #286
doctorpangloss wrote: post #241

Earlier quoted context omitted.

i'm not saying they reflect how much you study. they reflect a lot of things, including that. but you ask the people who write the tests, they're going to say, how much you know or how much you study, but nonetheless, they are limited. i agree with you. that's part of my point. let's imagine a different study. we instead compare AI-users and non-users on a Wechsler (IQ-adjacent) test. overall, it would be surprising…

I just started teaching undergrad CS courses after ~20 years of various non-academic jobs and it took me about two semesters to realize that almost nothing matters except how I assess students. Two weeks of ADHD-fueled research later, I concluded that academia is actively resistant to implementing assessment reform because it would expose the utter pointlessness of most of what happens in university classrooms. The r…

> almost nothing matters except how I assess students

This is a reasonable opinion after two semesters.

IMO, with more experience you should have modified your stance.

If all classes were pointless, students would leave at the same point they entered. Since we observe that is not the case, something is happening in class. Do more of that.

Re: AI boosted homework scores, then exam scores dropped: study

#380

Post by andy_ppp »

I was thinking how tricky it would be to learn to program now, honestly a sense of laziness is one of the reasons I learned programming - so that I could automate away repetitive tasks. Our generation might be the last people who can actually program properly, everyone will just learn some basic architecture and point the AI in vaguely the right direction.
Post reply on HN