11–20 of 41 posts
Earlier quoted context omitted.
The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. 1. Provide students with the tools and knowledge to critically verify responses, either coming from an educator or a an AI agent. 2. Build more transparent AI agents that show how reliable they are on different types of queries. Our deployment showed that the Help Fix Code w…
i agree with this. I keep trying to instill paranoia in the younger people I work with. even if you can see that the code is doing set_x(5), if it's crashing 20 lines down, I want you to either print or breakpoint the code here and really prove to me that x is now 5, before I look any further. sometimes set_x() might not do what you think. other times there might be something stomping on it from here to there, but I…
The way a formal methods lecturer explained to me his concerns about the Y2K problem by talking about the embedded systems in the automated medication pumps treating his sick partner, and how without an MMU and code that could not be inspected, there was a non-zero chance that rolled-over dates would cause logging data to overwrite configuration data?
Can an LLM convey a bit of anger and fear when talking about Therac-25?
Even though a TA is often at a much lower teaching level than this, every single person who has ever learned anything has done so with the benefit of a teacher who "got through to them" either on a topic or on a principle.
It's bonkers to compare TAs and LLMs simply on their error rate, when the errors TAs make are of a _totally_ different nature to the errors LLMs can make.
Did these solutions scores get penalized for lack of real understanding? Or, to put it another way, is your class about teaching programming itself or about teaching how to solve problems using any tool available (including an AI that solve it for you)?
Earlier quoted context omitted.
The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. 1. Provide students with the tools and knowledge to critically verify responses, either coming from an educator or a an AI agent. 2. Build more transparent AI agents that show how reliable they are on different types of queries. Our deployment showed that the Help Fix Code w…
> The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. Ehh. Those TAs, if they feel they might be wrong, can consult the lecturer/professor. And if they feel they might be wrong, they can just say so. IMO there is little to no comparison between a bad TA and a confidently-wrong LLM (having been a TA who knew to consult the pro…
We need both humans and AI. But there are problems with both, so that's why they can hopefully complement each other. Humans might have limited patience, availability, etc. and AI lacks empathy, and can be over-confident.
> Why are we doing this to academia when the better approach would be giving TAs better training in actual teaching?
Sure, that is a fantastic idea and some researchers have explored it.
But, what's wrong with doing exploratory research, in a real-world deployment? In the paper we describe both where CodeAid failed and where students and educators found it useful, in a very honest way.
> Direct code solution queries (44%) where students asked CodeAid to generate the direct solution (by copying the task description of their assignment). Did these solutions scores get penalized for lack of real understanding? Or, to put it another way, is your class about teaching programming itself or about teaching how to solve problems using any tool available (including an AI that solve it for you)?
Earlier quoted context omitted.
> The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. Ehh. Those TAs, if they feel they might be wrong, can consult the lecturer/professor. And if they feel they might be wrong, they can just say so. IMO there is little to no comparison between a bad TA and a confidently-wrong LLM (having been a TA who knew to consult the pro…
LLMs are tools. They're not everything. Yes, they can't sympathize or empathize. But if they can help a student to be more productive and learn at the same time, then I'm all in for designing them properly to be used in such educational contexts... "as an additional tool." We need both humans and AI. But there are problems with both, so that's why they can hopefully complement each other. Humans might have limited pa…
Genuine question: Why do we need both humans and AI? What's the evidence base for this statement?
I feel this is another thing that proponents state as if it's unchallengeable fact, an all-progress-is-good thing.
I question this assertion. People have become all too comfortable with it.
(Personal opinion: I don't think teaching needs AI at all, and if it does, a traditional simple expert system with crafted answers would still be better. I think there's a staggering range of opportunities for improving teaching materials that don't involve LLMs, and they are all being ignored because of where the hot money goes.)
Earlier quoted context omitted.
i agree with this. I keep trying to instill paranoia in the younger people I work with. even if you can see that the code is doing set_x(5), if it's crashing 20 lines down, I want you to either print or breakpoint the code here and really prove to me that x is now 5, before I look any further. sometimes set_x() might not do what you think. other times there might be something stomping on it from here to there, but I…
Right. So can an LLM convey that paranoia? The way a formal methods lecturer explained to me his concerns about the Y2K problem by talking about the embedded systems in the automated medication pumps treating his sick partner, and how without an MMU and code that could not be inspected, there was a non-zero chance that rolled-over dates would cause logging data to overwrite configuration data? Can an LLM convey a bit…
Earlier quoted context omitted.
Right. So can an LLM convey that paranoia? The way a formal methods lecturer explained to me his concerns about the Y2K problem by talking about the embedded systems in the automated medication pumps treating his sick partner, and how without an MMU and code that could not be inspected, there was a non-zero chance that rolled-over dates would cause logging data to overwrite configuration data? Can an LLM convey a bit…
oh my point was that somebody has to strike the fear of god in them first before they start trusting the llm blindly. I know the llm can fake this kind of thing, especially if you put a prompt that forces "as an LLM, I'm probably going to shoot you in the foot randomly", but I'm sure they'll get used to ignoring it.
We agree on that :-)
Earlier quoted context omitted.
What do you think about the ethical implications of using unreliable agents as educators?
The same goes with human TAs that are extensively used in undergrad introductory programming classes. They can also be unreliable in many cases. 1. Provide students with the tools and knowledge to critically verify responses, either coming from an educator or a an AI agent. 2. Build more transparent AI agents that show how reliable they are on different types of queries. Our deployment showed that the Help Fix Code w…
Earlier quoted context omitted.
LLMs are tools. They're not everything. Yes, they can't sympathize or empathize. But if they can help a student to be more productive and learn at the same time, then I'm all in for designing them properly to be used in such educational contexts... "as an additional tool." We need both humans and AI. But there are problems with both, so that's why they can hopefully complement each other. Humans might have limited pa…
> We need both humans and AI. Genuine question: Why do we need both humans and AI? What's the evidence base for this statement? I feel this is another thing that proponents state as if it's unchallengeable fact, an all-progress-is-good thing. I question this assertion. People have become all too comfortable with it. (Personal opinion: I don't think teaching needs AI at all, and if it does, a traditional simple expert…