Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CodeAid: A classroom deployment of an LLM-based coding assistant

austinhenley.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#31

Post by throw46365 »

majeedkazemi wrote: post #8
lispisok wrote: post #6

Earlier quoted context omitted.

My experience as a TA is students definitely do not have the knowledge to critically verify responses.

Doesn't this make AI agents better? Given that human TAs do make mistakes a LOT, or in many cases are just unprepared (e.g. haven't done the programming assignment themselves) Human TA's have ego, AI doesn't. With proper tools, you should be able to steer an AI agent. I think both humans and AI agents both have their drawbacks and benefits. That's why the last section of the paper discusses that we even need to teach…

Humans have their drawbacks?

I have to say, I really urge you to consider the nature of your framing here.

As you can tell I am quietly (or maybe not-so-quietly) appalled by the zeitgeist around generative AI, but even if I were not, I hope I would still see that this kind of linguistic framing is insensitive and self-defeating if not wholly inappropriate and demeaning if you want to see any co-operation from educators, who are as a broad picture, tired, dedicated, hopeful people who -- unlike LLMs -- can and do place significant moral and ethical value on teaching.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#32

Post by nucrow »

throw46365 wrote: post #31
majeedkazemi wrote: post #8

Earlier quoted context omitted.

Doesn't this make AI agents better? Given that human TAs do make mistakes a LOT, or in many cases are just unprepared (e.g. haven't done the programming assignment themselves) Human TA's have ego, AI doesn't. With proper tools, you should be able to steer an AI agent. I think both humans and AI agents both have their drawbacks and benefits. That's why the last section of the paper discusses that we even need to teach…

Humans have their drawbacks ? I have to say, I really urge you to consider the nature of your framing here. As you can tell I am quietly (or maybe not-so-quietly) appalled by the zeitgeist around generative AI, but even if I were not, I hope I would still see that this kind of linguistic framing is insensitive and self-defeating if not wholly inappropriate and demeaning if you want to see any co-operation from educat…

I'm curious as to how you would frame it instead.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#33

Post by nucrow »

throw46365 wrote: post #10
majeedkazemi wrote: post #4

Earlier quoted context omitted.

The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. 1. Provide students with the tools and knowledge to critically verify responses, either coming from an educator or a an AI agent. 2. Build more transparent AI agents that show how reliable they are on different types of queries. Our deployment showed that the Help Fix Code w…

> The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. Ehh. Those TAs, if they feel they might be wrong, can consult the lecturer/professor. And if they feel they might be wrong, they can just say so. IMO there is little to no comparison between a bad TA and a confidently-wrong LLM (having been a TA who knew to consult the pro…

Imagine we could, with the snap of a finger, come up with an AI tutor that is objectively better than human TAs. Better as in: between 2 groups of 10,000 students, those with AI tutors do better on perfomance metrics 95% of the time than those students with human tutors. Would you be opposed to replacing human tutors then?

If your answer is yes, (in some flavor of "protecting and helping the jobs of those who teach), I would argue your ethics are focused on the wrong group. Teaching is for students to learn, not for teachers to have jobs.

We don't have said technology yet, but it's reasonable to think we can get close. If there's a good chance to improve how well students can learn, I don't think "teachers don't appreciate it" is a good reason not to do it.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#34

Post by throw46365 »

nucrow wrote: post #32
throw46365 wrote: post #31

Earlier quoted context omitted.

Humans have their drawbacks ? I have to say, I really urge you to consider the nature of your framing here. As you can tell I am quietly (or maybe not-so-quietly) appalled by the zeitgeist around generative AI, but even if I were not, I hope I would still see that this kind of linguistic framing is insensitive and self-defeating if not wholly inappropriate and demeaning if you want to see any co-operation from educat…

I'm curious as to how you would frame it instead.

I wouldn't put myself in that position.

When the question is of the form "what do you think about the ethical implications of using AIs?" and the quick answer is "humans also make mistakes" I think the entire premise is on pretty shaky ground.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#35

Post by nucrow »

majeedkazemi wrote: post #2

I'm the lead author of this paper. Feel free to ask me anything! - MK

First off, congrats! Cool paper! Q: did you measure the correlation between CodeAid use and test scores? When educational youtube videos took off, people liked them a lot, but turns out people weren't really retaining very much information. You'd think this wouldn't be the case for CodeAid, but it's important to check

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#36

Post by throw46365 »

nucrow wrote: post #33
throw46365 wrote: post #10

Earlier quoted context omitted.

> The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. Ehh. Those TAs, if they feel they might be wrong, can consult the lecturer/professor. And if they feel they might be wrong, they can just say so. IMO there is little to no comparison between a bad TA and a confidently-wrong LLM (having been a TA who knew to consult the pro…

Imagine we could, with the snap of a finger, come up with an AI tutor that is objectively better than human TAs. Better as in: between 2 groups of 10,000 students, those with AI tutors do better on perfomance metrics 95% of the time than those students with human tutors. Would you be opposed to replacing human tutors then? If your answer is yes, (in some flavor of "protecting and helping the jobs of those who teach),…

> Would you be opposed to replacing human tutors then?

Yes.

> If your answer is yes, (in some flavor of "protecting and helping the jobs of those who teach), I would argue your ethics are focused on the wrong group. Teaching is for students to learn, not for teachers to have jobs.

OK. But your framing here projects upon me the idea that I'm solely concerned about replacing jobs, in order for your argument to succeed. (Though again, that is the cold-rationalist AI zeitgeist[0]: why should people have jobs when an AI can do it?)

It elides the possibility that it is inherently better to learn from a real person, who has invested time and effort into teaching you. What is the point of higher education in particular if you are not learning, at some point, from people who are directly adjacent to cutting edge thinking?

> We don't have said technology yet, but it's reasonable to think we can get close. If there's a good chance to improve how well students can learn, I don't think "teachers don't appreciate it" is a good reason not to do it.

Well then. I can't argue with this, if you think it's OK to take humanity out of teaching. I think — perhaps feel — you are so wrong that I can barely even string the words together to explain. And that is an unbridgeable divide.

[0] https://www.theatlantic.com/technology/archive/2024/05/opena... or https://archive.ph/AL81B

'In response to one question about AGI rendering jobs obsolete, Jeff Wu, an engineer for the company, confessed, “It’s kind of deeply unfair that, you know, a group of people can just build AI and take everyone’s jobs away, and in some sense, there’s nothing you can do to stop them right now.” He added, “I don’t know. Raise awareness, get governments to care, get other people to care. Yeah. Or join us and have one of the few remaining jobs. I don’t know; it’s rough.”'

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#37

Post by majeedkazemi »

nucrow wrote: post #35
majeedkazemi wrote: post #2

I'm the lead author of this paper. Feel free to ask me anything! - MK

First off, congrats! Cool paper! Q: did you measure the correlation between CodeAid use and test scores? When educational youtube videos took off, people liked them a lot, but turns out people weren't really retaining very much information. You'd think this wouldn't be the case for CodeAid, but it's important to check

Thanks!

> did you measure the correlation between CodeAid use and test scores?

Unfortunately, we couldn't measure such correlations due to many external factors that can impact students' performance on test scores.

However, our previous research specifically compared students learning python coding for the first time with and without LLM code generators. Austin and I wrote a summary on it here: https://austinhenley.com/blog/learningwithai.html

We found that students who performed higher on our Scratch programming pre-tests (before starting the Python lessons), performed significantly better if they had access to the code generator. However, the system that we used in that study was very different, it showed the exact code solution to the students' query. Here with CodeAid we were trying to avoid producing direct code solutions!

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#38

Post by nucrow »

throw46365 wrote: post #36
nucrow wrote: post #33

Earlier quoted context omitted.

Imagine we could, with the snap of a finger, come up with an AI tutor that is objectively better than human TAs. Better as in: between 2 groups of 10,000 students, those with AI tutors do better on perfomance metrics 95% of the time than those students with human tutors. Would you be opposed to replacing human tutors then? If your answer is yes, (in some flavor of "protecting and helping the jobs of those who teach),…

> Would you be opposed to replacing human tutors then? Yes. > If your answer is yes, (in some flavor of "protecting and helping the jobs of those who teach), I would argue your ethics are focused on the wrong group. Teaching is for students to learn, not for teachers to have jobs. OK. But your framing here projects upon me the idea that I'm solely concerned about replacing jobs, in order for your argument to succeed.…

> It elides the possibility that it is inherently better to learn from a real person, who has invested time and effort into teaching you.

I did elude it, for the sake of the argument. If it turns out that in fact human tutoring is fundamentally better, then there's of course no point in using an inferior system (sweeping accesibility and other concerns under the rug). Go humans, if we're better!

> What is the point of higher education in particular if you are not learning, at some point, from people who are directly adjacent to cutting edge thinking?

For the subset that do research, this matters a lot. But for most everyone else looking for a better job, it's not really relevant.

> Well then. I can't argue with this, if you think it's OK to take humanity out of teaching. I think — perhaps feel — you are so wrong that I can barely even string the words together to explain. And that is an unbridgeable divide

I appreciate your candidness, and perhaps it's true that we may just not be able to agree. For what it's worth, my bet is tutors' quality will improve, rather than them getting displaced. My point however is: I want my kid to learn as best as possible. If that turns out to be with a robot, I'm not making my kid worse off to save some guy's job.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#39

Post by nucrow »

throw46365 wrote: post #34
nucrow wrote: post #32

Earlier quoted context omitted.

I'm curious as to how you would frame it instead.

I wouldn't put myself in that position. When the question is of the form "what do you think about the ethical implications of using AIs?" and the quick answer is "humans also make mistakes" I think the entire premise is on pretty shaky ground.

I think I may have been unclear.

>... I would still see that this kind of linguistic framing is insensitive and self-defeating if not wholly inappropriate and demeaning if you want to see any co-operation from educators

I think this was spot on, and I agree with you. I meant to ask: have you thought about a way to frame it, in which educators don't feel threatened but instead excited ? Given that, from my perspective, there's a great chance to enhance both the teaching and learning experience, as well as results

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#40

Post by nucrow »

majeedkazemi wrote: post #37
nucrow wrote: post #35

Earlier quoted context omitted.

First off, congrats! Cool paper! Q: did you measure the correlation between CodeAid use and test scores? When educational youtube videos took off, people liked them a lot, but turns out people weren't really retaining very much information. You'd think this wouldn't be the case for CodeAid, but it's important to check

Thanks! > did you measure the correlation between CodeAid use and test scores? Unfortunately, we couldn't measure such correlations due to many external factors that can impact students' performance on test scores. However, our previous research specifically compared students learning python coding for the first time with and without LLM code generators. Austin and I wrote a summary on it here: https://austinhenley.c…

> Unfortunately, we couldn't measure such correlations due to many external factors that can impact students' performance on test scores.

Fair. Guess we'll have to roll it out to increase that sample size :)

> We found that students who performed higher on our Scratch programming pre-tests (before starting the Python lessons), performed significantly better if they had access to the code generator.

Interesting!

From the paper > Additionally, students in the Codex group were more eager and excited to continue learning about programming, and felt much less stressed and discouraged during the training

This by itself could reap great benefits years down the line. Me and my wife have felt the same way at work!

Post reply on HN