Accumulation of cognitive debt when using an AI assistant for essay writing task

Post by kibibu » Tue, Jun 17, 2025, 9:08 AM UTC

mjburgess wrote: post #197

sesm wrote: post #183

Earlier quoted context omitted.

One of the effects on software development is: the fact that you submitted a PR with any LoC count doesn't mean that you did any work. You need to explain your solution and answer questions to prove that.

The next stage of this issue is: how do you explain something you didn't write? The LLM-optimist view at the moment, which takes on board the need to review LLMs, assumes that this review capability will exist. I cannot review LLM output on areas outside of my expertise. I cannot develop the expertise I need if I use an LLM in-the-large. I first encountered this issue ~year-ago when using an LLM to prototype a progra…

Post by mjburgess » Tue, Jun 17, 2025, 9:15 AM UTC

Ekaros wrote: post #192

mjburgess wrote: post #179

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to…

> I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people Now one like me might go and ask how much of communication is actually worthwhile? Sometimes I consider that there is lot of communication that might not actually be. It is still done, but if no one actually reads it, why not automate generation. Not to say there is not significant amount of stuff you…

Post by falcor84 » Tue, Jun 17, 2025, 9:16 AM UTC

bayindirh wrote: post #176

Magmalgebra wrote: post #175

Well... yes? Essays are tools to force students to structure and communicate thinking - production of the essay forces the thinking. If you want an equivalent result from LLMs you're going to need a much more iterative process of critique and iteration to get the same kind of mental effort out of students. We haven't designed that process yet.

I mean, they found brain atrophy. If this doesn't get someone worried, I don't know what would. I joked that "I don't do drugs" when someone asked me whether I played MMORPGs, but this joke becomes just too real when we apply it to generative AI of any soırt.

Post by eru » Tue, Jun 17, 2025, 9:18 AM UTC

jonplackett wrote: post #182

I think we need to shift our idea of what LLMs do and stop thinking they are ‘thinking’ in any human way. The best mental description I have come up with is they are “Concept Processors”. Which is still awesome. Computers couldn’t understand concepts before. And now they can, and they can process and transform them in really interesting and amazing ways. You can transform the concept of ‘a website that does X’ into c…

Post by eru » Tue, Jun 17, 2025, 9:18 AM UTC

bayindirh wrote: post #176

Magmalgebra wrote: post #175

Well... yes? Essays are tools to force students to structure and communicate thinking - production of the essay forces the thinking. If you want an equivalent result from LLMs you're going to need a much more iterative process of critique and iteration to get the same kind of mental effort out of students. We haven't designed that process yet.

I mean, they found brain atrophy. If this doesn't get someone worried, I don't know what would. I joked that "I don't do drugs" when someone asked me whether I played MMORPGs, but this joke becomes just too real when we apply it to generative AI of any soırt.