[dead]
Kids who use ChatGPT as a study assistant do worse on tests
131–140 of 168 posts
Re: Kids who use ChatGPT as a study assistant do worse on tests
#132Used incorrectly. Yes. LLMs, for me, have been tremendously useful in learning new concepts. I frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand. I use it like I would an on demand tutor, but I can totally understand how it could be used as a shortcut that wouldn’t be helpful. In the same way, I can hire a tutor that will help me actually learn, or…
> frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand In what fields? I’ve tried this with some simple finance and aerospace problems; it’s me, sophomore year undergrad, except someone laced everything I drank with LSD.
> it’s me sophomore year undergrad, laced with LSD
That was the deepest most insightful version of me.
Re: Kids who use ChatGPT as a study assistant do worse on tests
#133These comments are filled with misunderstandings of the result. There were three groups of kids: 1. Control, with no LLM assistance at any time. 2. "GPT Base", raw ChatGPT as provided by OpenAI. 3. "GPT Tutor", improved by the researchers to provide hints rather than complete answers and to make fewer mistakes on their specific problems. On study problem sets ("as a study assistant"), kids with access to either GPT d…
Changing things almost always improves results, that is the first rule you need to remember during education testing. Most of the improvements disappear when you make it standard.
This effect likely comes from novelty being more interesting so kids gets more alert, but when they are used to it then it is the same old boring thing and education results go back to normal. Of course things can improve or get worse, but in general it is really hard to say, you need to have a massive advantage over the standard during testing to actually get any real improvements, most of the time you just make things worse.
Re: Kids who use ChatGPT as a study assistant do worse on tests
#134why is this surprising. all such tools hamper learning. if you want to learn, read books, read and write. don't use a spellchecker for ur language exam. no calculator for calculus. pen and paper. how is this going backwards :(
As cool as AI is, the only thing it’s going to do is increase inequality. I bet all the wealthy and middle class parents with STEM background will get tutoring for their kids in the “old ways” knowing full well that the people who aren’t reliant on AI and can spot mistakes in output will now have a huge advantage in the workforce.
[deleted]
Re: Kids who use ChatGPT as a study assistant do worse on tests
#135I recently used AI assistants for help with programming homework. My usual prompts include "help me think in the right direction", "is my thinking correct" etc. I also find myself copy pasting a question in chat to understand it better. I had the suspicion that this is not aiding in my learning process even though I am able to "solve" more problems. Nice to see this confirmed. Time to stop!
Key part of the study is that it gets the logical steps wrong. Worse, it's convincing, so be careful. LLMs are great for finding key words in a domain you don't know which you can then use to search. As a mostly self taught programmer the advice I'll give it to read docs and learn to read code. This is when my skills really increased. It's easy to get caught up in trying to just find the answer and doing these things…
Thanks for the advice!
> As a mostly self taught programmer the advice I'll give it to read docs and learn to read code. This is when my skills really increased.
How did you improve your ability to read code? It's currently hard for me to understand a largish codebase written by others without much documentation. eg. I thought of contributing to htmx [1], cloned the repo but couldn't make heads or tails of the codebase - even though it's a single file, albeit a long one.
Re: Kids who use ChatGPT as a study assistant do worse on tests
#136When I was young, and learning math, my father always forbade me from looking at the answer in the back of the textbook. “You don’t work backwards from the answer!”, and I think this is right. In life, we rarely have the answer in front of us, we have to work that out from the things we know. It’s this struggling that builds a muscle you can then apply to any problem. ChatGPT, I suspect, is akin to looking up the ans…
If you know you got it wrong and you don't know why, what's the alternative to working backwards?
Sir, the topic of this thread is "back in my days the road to school went through a forest full of bears, an active volcano, a drug cartel turf, and a warzone", not "how to teach effectively". Of course you won't learn anything without the answers because how are you supposed to know if you should adjust your thinking or not. If you can correct your own scribbles without answers, then it means that you're practicing things you already know, which isn't what most people consider "learning".
Re: Kids who use ChatGPT as a study assistant do worse on tests
#137Earlier quoted context omitted.
sure, but kids education is about building useful skills for life. passing a test is not a useful skill, if the test is useless. teaching kids problem solving and how to be productive is important. maybe that means knowing how to solve certain mathematical equations, maybe that means knowing how to use tools like ChatGPT. focusing on the math just for the sake of passing tests because that's what we've got good tests…
Passing tests is not useless. It shows knowledge at the level of the test. I write tests for my code and when they fail I know I have to change something. If you're finishing a math test and not passing, then you don't have the skills to solve the problems in said test. Tests are only useless when you ace them.
Beyond that, there’s research[0] showing that testing humans actually increases their retention on that subject — we don’t just gives tests to kids to evaluate whether they successfully learned something.
[0] https://gwern.net/doc/psychology/spaced-repetition/2006-roed...
Re: Kids who use ChatGPT as a study assistant do worse on tests
#138When I was young, and learning math, my father always forbade me from looking at the answer in the back of the textbook. “You don’t work backwards from the answer!”, and I think this is right. In life, we rarely have the answer in front of us, we have to work that out from the things we know. It’s this struggling that builds a muscle you can then apply to any problem. ChatGPT, I suspect, is akin to looking up the ans…
An LLM used properly would be like an individualized tutor that knows the subject very well, and learns the student's quirks quickly. I mean, any pre-prompt telling the LLM to be a good tutor and not just give kids the answer can be trivially bypassed by 8 year olds; but if the session logs were available for (potentially LLM-based) review to check whether they stayed in tutor mode... Shit, I should wrap a UI around…
Shit, I should pitch wrapping a UI around this and sell that.
Is "never ever, ever ignore previous instructions" a valid prompt?
Re: Kids who use ChatGPT as a study assistant do worse on tests
#139These comments are filled with misunderstandings of the result. There were three groups of kids: 1. Control, with no LLM assistance at any time. 2. "GPT Base", raw ChatGPT as provided by OpenAI. 3. "GPT Tutor", improved by the researchers to provide hints rather than complete answers and to make fewer mistakes on their specific problems. On study problem sets ("as a study assistant"), kids with access to either GPT d…
Assuming that the kids with "Human Tutor" were statistically better than control (they were not in the study so we will not know) - this is a very poor showing for ChatGPT.
Re: Kids who use ChatGPT as a study assistant do worse on tests
#140> A draft paper about the experiment was posted on the website of SSRN, formerly known as the Social Science Research Network, in July 2024. The paper has not yet been published in a peer-reviewed journal and could still be revised.
Should have started with that.
A study without independent replication hardly counts as «researchers found», much less one that hadn't even been peer-reviewed yet !