[deleted]
Kids who use ChatGPT as a study assistant do worse on tests
21–30 of 168 posts
Re: Kids who use ChatGPT as a study assistant do worse on tests
#22From the abstract:
“Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base).”
Kids who use ChatGPT do actually “significantly” better according to the authors. Now I don’t know if significantly means statistically significant here because I haven’t read the methodology but 127% increase in performance must be something. That said, that’s a clickbaity title if I’ve ever seen one.
Edit: Upon closer reading, the increase in performance is statistically significant. Also “access to GPT“ in this case is having GPT open while solving the problems, not studying with GPT and then solving the problems, which was my first understanding from the clickbaity title. Results are not terribly surprising in that regard.
Re: Kids who use ChatGPT as a study assistant do worse on tests
#23What were the primary reasons that made students who used ChatGPT do poorly on math assessments, even though they had worked correctly through a greater number of practice problems?
They didn’t have to figure out how to solve the problem. Instead of struggling a bit, which is where the learning happens, they would likely go to ChatGPT for the answer. When the answer bot was taken away, they weren’t prepared to think about how to solve the problem and work it out.
I’ve noticed this even using Copilot in VS Code. I rarely use it, but if I start pulling it out, I notice at the first hint of actually thinking about how to do something, my brain seeks to ask Copilot instead. It’s like there is an off switch that gets flicked when there is an easy button available. If I were to figure it out on my own, I’d know what to do next time I run into a problem like that… if I use Copilot, I’ve learned nothing, other than next time I run into this, use Copilot.
It’s a crutch when it comes to learning.
Re: Kids who use ChatGPT as a study assistant do worse on tests
#24I recently used AI assistants for help with programming homework. My usual prompts include "help me think in the right direction", "is my thinking correct" etc. I also find myself copy pasting a question in chat to understand it better.
I had the suspicion that this is not aiding in my learning process even though I am able to "solve" more problems. Nice to see this confirmed. Time to stop!
Re: Kids who use ChatGPT as a study assistant do worse on tests
#25What if the test is irrelevant to the current times?
“Those with ChatGPT solved 48 percent more of the practice problems correctly, but they ultimately scored 17 percent worse on a test of the topic that the students were learning.”
So, in the real world, where people can use chatgpt in their jobs, the kids that use it will do better than the kids who don’t.
Maybe a better test is: can you catch chatgpt when it is wrong? Not, can you answer without ChatGPT?
Re: Kids who use ChatGPT as a study assistant do worse on tests
#26Why is the study specifically of Turkish students
The paper at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4895486 says:
> One of the co-authors, Özge Kabakcı, a high school math teacher and former department chair of the math department at our partner Turkish high school, led the development of all session materials.
Re: Kids who use ChatGPT as a study assistant do worse on tests
#27Why is the study specifically of Turkish students
I guess it would be hard to find a high school with the sample size that you need (thousand) that will agree on collaborating. And in the US every county will have different rules and in terms of math they don't teach it in standard way.
But why Turkish not British or any other place is going to be a question no matter the location. But do you really think the results will be significantly different if it is done lets say on Vietnamese students?
Re: Kids who use ChatGPT as a study assistant do worse on tests
#28> A third group of students had access to a revised version of ChatGPT that functioned more like a tutor. This chatbot was programmed to provide hints without directly divulging the answer. The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids.
Is it me, or is does this directly contradicts the title?
Re: Kids who use ChatGPT as a study assistant do worse on tests
#29From the abstract: “Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base).” Kids who use ChatGPT do actually “significantly” better according to the a…
You can read the paper itself to get my useful information. The title and article is confusing.
Re: Kids who use ChatGPT as a study assistant do worse on tests
#30From the abstract: “Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base).” Kids who use ChatGPT do actually “significantly” better according to the a…
They don't do better as far as learning is concerned. Isn't this concerning?
> However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base).
The abstract continues:
> That is, access to GPT-4 can harm educational outcomes. These negative learning effects are largely mitigated by the safeguards included in GPT Tutor. Our results suggest that students attempt to use GPT-4 as a "crutch" during practice problem sessions, and when successful, perform worse on their own. Thus, to maintain long-term productivity, we must be cautious when deploying generative AI to ensure humans continue to learn critical skills.