51–60 of 168 posts
When I was young, and learning math, my father always forbade me from looking at the answer in the back of the textbook. “You don’t work backwards from the answer!”, and I think this is right. In life, we rarely have the answer in front of us, we have to work that out from the things we know. It’s this struggling that builds a muscle you can then apply to any problem. ChatGPT, I suspect, is akin to looking up the ans…
Kids who have their parents do their homework do worse on tests. s/parents/chatgpt
Your characterization of "do their homework" seems wrong as the paper says of the GPT Tutor branch that "is hard for students to use it as a crutch since its prompt asks it to avoid giving them the answer and instead guide them in a step-by-step fashion."
From the abstract: “Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base).” Kids who use ChatGPT do actually “significantly” better according to the a…
If this is your takeaway you misread the paper. Students have access to GPT (if they have access, the control didn't) while working through practice problems. Not for the exam itself. From the paper in the experimental design section:
> Each session has three parts:
> 1. In the first part, teachers review a topic (e.g., combinatorics) previously covered in the course, and solve one or more examples on the board. This part is identical to a standard high school one-to-many (i.e., teacher-to-students) lecture.
> 2. The second part is an assisted practice period, where students solve a sequence of exercises designed by teachers to reinforce the covered concept. Our randomized intervention (described in more detail below) only affects this second, self-study part.
> 3. The third part is an unassisted evaluation, where students take a closed-book, closed laptop exam. Importantly, each problem in the exam corresponds to a conceptually very similar practice problem from the previous part—this design was chosen to help students practice the key concepts needed to perform well on the exam.
Students with GPT (either form) did better during the practice problem portion and then worse during the actual exam (without GPT access) than students in the control.
When I was young, and learning math, my father always forbade me from looking at the answer in the back of the textbook. “You don’t work backwards from the answer!”, and I think this is right. In life, we rarely have the answer in front of us, we have to work that out from the things we know. It’s this struggling that builds a muscle you can then apply to any problem. ChatGPT, I suspect, is akin to looking up the ans…
If you know you got it wrong and you don't know why, what's the alternative to working backwards?
Used incorrectly. Yes. LLMs, for me, have been tremendously useful in learning new concepts. I frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand. I use it like I would an on demand tutor, but I can totally understand how it could be used as a shortcut that wouldn’t be helpful. In the same way, I can hire a tutor that will help me actually learn, or…
In what fields? I’ve tried this with some simple finance and aerospace problems; it’s me, sophomore year undergrad, except someone laced everything I drank with LSD.
From the abstract: “Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base).” Kids who use ChatGPT do actually “significantly” better according to the a…
People sophisticated in a field can ask Sonnet or 4o questions that amount to a different way of searching and sometimes even a better one. If you ask a question in a direct, probing, narrow way you can sometimes come out ahead. Someone educated by the News Feed algorithm (which is what RLHF amounts to: reward for getting human to click) is going to be the worst kind of wrong: /r/ConfidentlyIncorrect.
PS was there ever a blog at b7r6.net?
[flagged]
> A third group of students had access to a revised version of ChatGPT that functioned more like a tutor. This chatbot was programmed to provide hints without directly divulging the answer. The students who used it did spectacularly better on the practice problems, solving 127 percent more of them correctly compared with students who did their practice work without any high-tech aids. Is it me, or is does this direct…
At best they do the same, at the worst they do worse. There was no scenario where ChatGPT actually helped (unless you count practice problems, which are apparently a meaningless stat, since it doesn’t translate to test scores).
Most kids in the general public would use normal ChatGPT, not one specially optimized to act like a tutor. So in the real world, kids who use it will do worse, and it’s not with seeking out the special one, as there is no advantage.
The title seems like a pretty fair 1 line summary, and pours some much needed cold water on a space that has been full of hype and promises, many of which have fallen flat.
If you take the measuring tape away from the person who relied on that tool instead of being good at using a meterstick, or perhaps no tools besides their own arm length, they are gonna suddenly not be able to measure, unless they go through the effort of learning to measure without the tape.
You can argue that measuring tape is a crutch preventing people from learning how to properly measure, and has its own limitations, but regardless its still really helpful, especially for people who only need to measure things occassionally, and not super long things.
ChatGPT is a tool. Just like all other tools, like computers, cars, etc., if you take it away, most people cannot perform the function for which they relied on the tool to help them do.