CodeAid: A classroom deployment of an LLM-based coding assistant • Hacker News Board

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#21

Post by tmpz22 » Sat, Jun 08, 2024, 9:12 PM UTC

camdenreslink wrote: post #19

majeedkazemi wrote: post #4

Earlier quoted context omitted.

The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. 1. Provide students with the tools and knowledge to critically verify responses, either coming from an educator or a an AI agent. 2. Build more transparent AI agents that show how reliable they are on different types of queries. Our deployment showed that the Help Fix Code w…

The TAs in my undergraduate intro to programming class were very knowledgable and reliable, but that is a sample size of 1.

The Grad student teachers and TAs in my math courses - including discrete math - were at best ambivalent to us lesser Computer Science students and at worst under-trained and contemptuous.

University of Oregon ~2014ish

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#22

Post by [deleted] » Sat, Jun 08, 2024, 9:18 PM UTC

majeedkazemi wrote: post #20

throw46365 wrote: post #16

Earlier quoted context omitted.

> We need both humans and AI. Genuine question: Why do we need both humans and AI? What's the evidence base for this statement? I feel this is another thing that proponents state as if it's unchallengeable fact, an all-progress-is-good thing. I question this assertion. People have become all too comfortable with it. (Personal opinion: I don't think teaching needs AI at all, and if it does, a traditional simple expert…

I think my stance is pretty clear about "utilizing" AI in educational settings. We absolutely don't need AI the same way we need air to breathe. But AI could potentially provide some solutions (and create new problems or have adverse effects as well), so why not explore it properly to find out where it works and where it doesn't?

[deleted]

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#23

Post by hombre_fatal » Sat, Jun 08, 2024, 9:38 PM UTC

jhp123 wrote: post #9

the explanation of "*word = "hello"" shown is completely incorrect, the memmove explanation is also incorrect

> It was developed by Majeed as a web app that uses GPT3.5 to power an assortment of AI features

Really need GPT-4 for technical explanations, but it's also much more expensive.

Every once in a while, ChatGPT logs me out and switches to GPT-3, and I immediately notice just due to the quality of the answer.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#24

Post by hombre_fatal » Sat, Jun 08, 2024, 9:45 PM UTC

throw46365 wrote: post #10

majeedkazemi wrote: post #4

Earlier quoted context omitted.

The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. 1. Provide students with the tools and knowledge to critically verify responses, either coming from an educator or a an AI agent. 2. Build more transparent AI agents that show how reliable they are on different types of queries. Our deployment showed that the Help Fix Code w…

> The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. Ehh. Those TAs, if they feel they might be wrong, can consult the lecturer/professor. And if they feel they might be wrong, they can just say so. IMO there is little to no comparison between a bad TA and a confidently-wrong LLM (having been a TA who knew to consult the pro…

Meanwhile almost every TA I had at uni didn't really want to be there. They were there for their PhD, not as a professor in training which would have made your position more understandable. And to boot they rarely spoke English very well. I had a few TAs that I understood so poorly that I stopped attending their labs.

The TA system feels like a hack where university gets to get free labor out of PhD students, but the undergrads suffer for it. I don't think there's much to glamorize. Nor do I think there's much to salvage from the days where you needed to attend office hours to get help. You see it as this critical human experience in uni but I don't.

That said, half my professors at uni also prob didn't want to teach. They were there for research.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#25

Post by throw46365 » Sat, Jun 08, 2024, 9:58 PM UTC

hombre_fatal wrote: post #24

throw46365 wrote: post #10

Earlier quoted context omitted.

> The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. Ehh. Those TAs, if they feel they might be wrong, can consult the lecturer/professor. And if they feel they might be wrong, they can just say so. IMO there is little to no comparison between a bad TA and a confidently-wrong LLM (having been a TA who knew to consult the pro…

Meanwhile almost every TA I had at uni didn't really want to be there. They were there for their PhD, not as a professor in training which would have made your position more understandable. And to boot they rarely spoke English very well. I had a few TAs that I understood so poorly that I stopped attending their labs. The TA system feels like a hack where university gets to get free labor out of PhD students, but the…

> They were there for their PhD, not as a professor in training which would have made your position more understandable.

Right. Not all TAs become professors. But at a first approximation all professors have TA experience; it's generally their first experience of teaching.

I was paid for my time as a TA, in the UK. It would be illegal for them not to pay.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#26

Post by batch12 » Sat, Jun 08, 2024, 10:40 PM UTC

majeedkazemi wrote: post #4

newzisforsukas wrote: post #3

Earlier quoted context omitted.

What do you think about the ethical implications of using unreliable agents as educators?

The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. 1. Provide students with the tools and knowledge to critically verify responses, either coming from an educator or a an AI agent. 2. Build more transparent AI agents that show how reliable they are on different types of queries. Our deployment showed that the Help Fix Code w…

> The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases

I think one difference is that human TAs can, theoretically, be held accountable for their reliability whereas a holding a LLM accountable is a little more difficult.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#27

Post by batch12 » Sat, Jun 08, 2024, 10:46 PM UTC

throw46365 wrote: post #16

majeedkazemi wrote: post #14

Earlier quoted context omitted.

LLMs are tools. They're not everything. Yes, they can't sympathize or empathize. But if they can help a student to be more productive and learn at the same time, then I'm all in for designing them properly to be used in such educational contexts... "as an additional tool." We need both humans and AI. But there are problems with both, so that's why they can hopefully complement each other. Humans might have limited pa…

> We need both humans and AI. Genuine question: Why do we need both humans and AI? What's the evidence base for this statement? I feel this is another thing that proponents state as if it's unchallengeable fact, an all-progress-is-good thing. I question this assertion. People have become all too comfortable with it. (Personal opinion: I don't think teaching needs AI at all, and if it does, a traditional simple expert…

The statement is a false statement to begin with. We don't have AI yet. Maybe when we have software that is truly intelligent, we can let it teach us. Until then I see this more as a buggy interactive textbook and agree with the author's description of it as a tool and disagree with the idea of it as a teacher.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#28

Post by eikenberry » Sat, Jun 08, 2024, 10:49 PM UTC

majeedkazemi wrote: post #15

eikenberry wrote: post #13

> Direct code solution queries (44%) where students asked CodeAid to generate the direct solution (by copying the task description of their assignment). Did these solutions scores get penalized for lack of real understanding? Or, to put it another way, is your class about teaching programming itself or about teaching how to solve problems using any tool available (including an AI that solve it for you)?

No. Students' usage was anonymized, so the course instructors did not know who used the system in what way. This was to make sure that students could use the tool freely without feeling like the instructors are watching their usage.

OK, cool. Hadn't realized it was designed as an experiment. So that would more be something the potential users might want to consider when they read about this. Thank you for clarifying.

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#29

Post by [deleted] » Sat, Jun 08, 2024, 10:56 PM UTC

majeedkazemi wrote: post #4

newzisforsukas wrote: post #3

Earlier quoted context omitted.

What do you think about the ethical implications of using unreliable agents as educators?

The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. 1. Provide students with the tools and knowledge to critically verify responses, either coming from an educator or a an AI agent. 2. Build more transparent AI agents that show how reliable they are on different types of queries. Our deployment showed that the Help Fix Code w…

[deleted]

Re: CodeAid: A classroom deployment of an LLM-based coding assistant

#30

Post by lispisok » Sun, Jun 09, 2024, 12:08 AM UTC

majeedkazemi wrote: post #8

lispisok wrote: post #6

Earlier quoted context omitted.

My experience as a TA is students definitely do not have the knowledge to critically verify responses.

Doesn't this make AI agents better? Given that human TAs do make mistakes a LOT, or in many cases are just unprepared (e.g. haven't done the programming assignment themselves) Human TA's have ego, AI doesn't. With proper tools, you should be able to steer an AI agent. I think both humans and AI agents both have their drawbacks and benefits. That's why the last section of the paper discusses that we even need to teach…

TA's can say things like "I dont know, lets figure it out together". LLMs will spit out false information as easily and confidently as true information.