Live data from Hacker News

Kids who use ChatGPT as a study assistant do worse on tests

hechingerreport.org

151–160 of 168 posts

Re: Kids who use ChatGPT as a study assistant do worse on tests

#151
post #19

Earlier quoted context omitted.

I can report that I have learned more about bash scripting in the past couple of years using ChatGPT to write all manner of scripts, than I did in the previous 20 years of copying and pasting things off StackOverflow.

You’re supposed to learn by reading books and doing exercises by yourself ( https://tldp.org/LDP/abs/html/ ). SO is equivalent to ChatGPT and will teach you nothing deep.

The difference is you can't query or chat with a book.

You can with a LLM. So if you are learning some thing new, you can keep poking and asking questions until you reach some level of understanding.

Im guessing with a book a lot of that has to happen with yourself.

Re: Kids who use ChatGPT as a study assistant do worse on tests

#152

Earlier quoted context omitted.

+1 insightful PS was there ever a blog at b7r6.net?

Not yet. I’m still grinding out what disclosure is and isn’t responsible. But bet your bum I’ve got a theme picked out. I didn’t realize anyone cared who I was enough to know I held the domain.

I was just curious; liked your comment, checked your hn profile, visited the domain it listed (shrug)

Re: Kids who use ChatGPT as a study assistant do worse on tests

#153
post #128

Earlier quoted context omitted.

Agreed.. All the co-pilot devs where I work are the juniors. Most of us just type the code we want.

Experts are often slower at starting to learn new tricks, because they don't need them, but are better at using them once they do. But for junior devs, it's often the blind leading the blind. That said, I don't actually use ChatGTP much. I'm just pretty sure I should in some areas.

Not a junior, but I find it useful.

Generating test data, adding documentation template for functions (which tbh is less useful when you specify types everywhere, even inferred types) and recently, after reading a HN comment, 'rubber ducking'. Which is great, because almost each time I need the rubber duck technique was for real obvious and easy issues I overcomplicated (last was induced by a naming change), and ChatGPT is quite good at that.

Re: Kids who use ChatGPT as a study assistant do worse on tests

#154
post #150

Earlier quoted context omitted.

As cool as AI is, the only thing it’s going to do is increase inequality. I bet all the wealthy and middle class parents with STEM background will get tutoring for their kids in the “old ways” knowing full well that the people who aren’t reliant on AI and can spot mistakes in output will now have a huge advantage in the workforce.

This can also back fire. I remember the days when people said programmers who don't use the internet/Google while coding actually learn more than those who do. While that was true initially- Eventually internet was just the norm and all pervasive. People who didn't use just got more unproductive, Same with companies. Sure grind on whatever you are learning, but don't equate suffering with making progress. After a whi…

This is an interesting thought. I learned programming thru a text editor vim. I didn’t have autocomplete and LSPs wouldn’t be invented until 8 years later. I wasn’t smart enough to figure out how to install/setup omnicomplete or snippets. All this meant I had to seek out the docs every time I wanted to use a method (or even be aware of an API), I had to write the same method every time.

I’d like to think this paid off for me, when writing JS I have a good heuristic on what methods do what that was slowly developed over constant repetition.

Now that I’m mentoring others I notice how others work with these modern editor features. It’s certainly faster than how I did it but I do wonder if there is a difference in aptitude.

When I worked at my first large corporation there was a very intelligent dev. I noticed over time he never really looked up answers on stack overflow or random blogs. He always went straight to the docs and the source code itself. I like to think his method of deliberate slowdown has paid off massively, even the way he asks questions was better than the rest of us.

It is hard to know which ways are better for learning and in the end we were all roughly making the same amount but there has to be something more to the usual pedagogy for software engineering.

I do wonder if tools like autocomplete, myriad of internet answers or musings, and now LLMs may be a hindrance for initial learning but it’s always hard to make these arguments because we have the benefit of learned experience whereas the new generation are now using different tools than us yet still arrive at the same conclusions.

Re: Kids who use ChatGPT as a study assistant do worse on tests

#155

Used incorrectly. Yes. LLMs, for me, have been tremendously useful in learning new concepts. I frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand. I use it like I would an on demand tutor, but I can totally understand how it could be used as a shortcut that wouldn’t be helpful. In the same way, I can hire a tutor that will help me actually learn, or…

> frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand In what fields? I’ve tried this with some simple finance and aerospace problems; it’s me, sophomore year undergrad, except someone laced everything I drank with LSD.

Technical fields.

Right now I'm going through the Dive Into Deep Learning course/textbook. Oftentimes it will gloss over a concept, or assume knowledge.

For a specific example: I wanted an expansion of why there are different loss functions and why you might choose one over another since the source material sort of plowed right through it. I got a good answer to my question without having to spend an hour reading through other materials

Re: Kids who use ChatGPT as a study assistant do worse on tests

#156

Used incorrectly. Yes. LLMs, for me, have been tremendously useful in learning new concepts. I frequently feed it my own notes and ask it to correct any misunderstandings, or to expand on things I don’t understand. I use it like I would an on demand tutor, but I can totally understand how it could be used as a shortcut that wouldn’t be helpful. In the same way, I can hire a tutor that will help me actually learn, or…

> LLMs, for me, have been tremendously useful in learning new concepts. Are you validating this in a rigorous way like this study, or are you just "feeling" like it's useful for your learning?

You want me to put a sample size of 1 anecdote through a peer reviewed study?

All I'm saying is that I find it to be useful to be able to ask questions about topics that I'm learning, and have them answered instantly instead of having to use a search engine or other resource to hunt down an answer to a specific question.

For a recent example where I am learning about deep learning: In what circumstances is MSE a useful loss function for regression? This is not a hard question to answer from a variety of resources, but I find it particularly useful to have my question answered differently instead of having to sort through a variety of resources. Maybe I'm losing something by not having to click through a few different resources and parse information that isn't directly relevant to what I want to learn. But for me, getting the answer fast keeps me from having to break away from the actual material that I'm learning.

Re: Kids who use ChatGPT as a study assistant do worse on tests

#157

From the abstract: “Consistent with prior work, our results show that access to GPT-4 significantly improves performance (48% improvement for GPT Base and 127% for GPT Tutor). However, we additionally find that when access is subsequently taken away, students actually perform worse than those who never had access (17% reduction for GPT Base).” Kids who use ChatGPT do actually “significantly” better according to the a…

> Kids who use ChatGPT do actually “significantly” better according to the authors

No, ChatGPT does significantly better than the kids who don't have access to ChatGPT.

Copy-pasting answers from ChatGPT isn't some amazing skill.

Re: Kids who use ChatGPT as a study assistant do worse on tests

#158
post #37

When I was young, and learning math, my father always forbade me from looking at the answer in the back of the textbook. “You don’t work backwards from the answer!”, and I think this is right. In life, we rarely have the answer in front of us, we have to work that out from the things we know. It’s this struggling that builds a muscle you can then apply to any problem. ChatGPT, I suspect, is akin to looking up the ans…

I think it depends on how the tool is used. If a student is just plugging in the problem and asking for the answer, there is clearly no long term benefit to this. If the student is trying to understand a concept, and uses GPT to bounce ideas around or to ask for alternate ways of thinking about something, it can be very helpful. How spacing is utilized can be helpful too. Struggle with the problem for 20-30 minutes.…

children using calculators to solve multiplication exercises do worse in multiplication exams

Ai is a tool. Use it as a tool - get benefits, use it to cheat...

Re: Kids who use ChatGPT as a study assistant do worse on tests

#159

Earlier quoted context omitted.

Key part of the study is that it gets the logical steps wrong. Worse, it's convincing, so be careful. LLMs are great for finding key words in a domain you don't know which you can then use to search. As a mostly self taught programmer the advice I'll give it to read docs and learn to read code. This is when my skills really increased. It's easy to get caught up in trying to just find the answer and doing these things…

Thanks for the advice! > As a mostly self taught programmer the advice I'll give it to read docs and learn to read code. This is when my skills really increased. How did you improve your ability to read code? It's currently hard for me to understand a largish codebase written by others without much documentation. eg. I thought of contributing to htmx [1], cloned the repo but couldn't make heads or tails of the codeba…

  > How did you improve your ability to read code? 
You're going to hate me for this, but it's by reading lots of code and being very very confused. When tackling a large code base I still pull out pen and paper and create flow charts.

And don't fret that you suck at it now. These are really hard things but they take time

Post reply on HN