Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accumulation of cognitive debt when using an AI assistant for essay writing task

arxiv.org

Post reply on HN
Print view View original

171–180 of 442 posts

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#171

Post by fhd2 »

greekanalyst wrote: post #170

"...the LLM group's participants performed worse than their counterparts in the Brain-only group at all levels: neural, linguistic, scoring." That's not surprising but also bleak.

Appears to align with good old Ironies of Automation [1]. If humans just review and rubber stamp results, they do a pretty terrible job at it.

I've been thinking for a while now that in order to truly make augmented workflows work, the mode of engagement is central. Reviewing LLM code? Bah. Having an LLM watch over my changes and give feedback? Different story. It's probably gonna be difficult and not particularly popular, but if we don't stay in the driver's seat somehow, I guess things will get pretty bleak.

[1]: https://en.m.wikipedia.org/wiki/Ironies_of_Automation

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#172

Post by kiru_io »

niemandhier wrote: post #23

AI is the anti-Zettelkasten. Rather than getting ever deeper insight into a subject matter by actively working on it, you iterate fast but shallow over a corpus of AI generated content. Example: I wanted to understand the situation in the Middle East better so I wrote an 10 page essay on the genesis if Hammas and Hizbulah using OpenAI as a cowriter. I remember nothing, worse of the things I remember I don’t know if i…

Interesting perspective to see AI as the opposite of accessing connected knowledge (aka Zettelkasten)

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#174

Post by falcor84 »

542354234235 wrote: post #156
falcor84 wrote: post #139

Earlier quoted context omitted.

> "LLMs are good at reducing text, not expanding it" You put it in quote marks, but the only search results are from you writing it here on HN. Obviously LLMs are extremely good at expanding text, which is essentially what they do whenever they continue a prompt. Or did you mean that in a prescriptive way - that it would be better for us to use it more for summarizing rather than expanding?

>You put it in quote marks, but the only search results are from you writing it here on HN. They said it was a rule of thumb, which is a general rule based on experience. In context with the comment they were replying to, it seems that they are saying that if you want to learn and understand something, you should put the effort in yourself first to synthesize your ideas and write out a full essay, then use an LLM to…

I think that this conflates two issues though. It seems obvious to me that in general, the more time and effort I put into a task, the deeper I will understand it. But it's unclear to me how this aspect of how we learn by spending time on a task is related to what LLMs are good at.

Intentionally taking this to a slightly absurd metaphor - it seemed to me like a person saying that their desire to reduce their alcohol consumption, led them to infer the rule of thumb that "waiters are good at bringing food, not drinks".

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#175

Post by Magmalgebra »

Well... yes? Essays are tools to force students to structure and communicate thinking - production of the essay forces the thinking. If you want an equivalent result from LLMs you're going to need a much more iterative process of critique and iteration to get the same kind of mental effort out of students. We haven't designed that process yet.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#176

Post by bayindirh »

Magmalgebra wrote: post #175

Well... yes? Essays are tools to force students to structure and communicate thinking - production of the essay forces the thinking. If you want an equivalent result from LLMs you're going to need a much more iterative process of critique and iteration to get the same kind of mental effort out of students. We haven't designed that process yet.

I mean, they found brain atrophy. If this doesn't get someone worried, I don't know what would.

I joked that "I don't do drugs" when someone asked me whether I played MMORPGs, but this joke becomes just too real when we apply it to generative AI of any soırt.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#177

Post by tuatoru »

fhd2 wrote: post #171
greekanalyst wrote: post #170

"...the LLM group's participants performed worse than their counterparts in the Brain-only group at all levels: neural, linguistic, scoring." That's not surprising but also bleak.

Appears to align with good old Ironies of Automation [1]. If humans just review and rubber stamp results, they do a pretty terrible job at it. I've been thinking for a while now that in order to truly make augmented workflows work, the mode of engagement is central. Reviewing LLM code? Bah. Having an LLM watch over my changes and give feedback? Different story. It's probably gonna be difficult and not particularly po…

Didn't realise the pedigree of the idea went back to 1983.

I read about this in a book "Our Robots, Ourselves". That talked about airline pilots' experience with auto-land systems introduced in the late 1990s/ early 2000s.

As you'd expect after having read Ironies of Automation, after a few near misses and not misses, auto-land is not used any more. Instead, pilot augmentation with head-up displays is used.

What is the programming equivalent of a head-up display?

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#178

Post by panstromek »

Interesting. This says a different thing than what I thought from the title. I thought this will be about cognitive overload from having to process and review all the text the LLM generates.

I had to disable copilot for my blog project in the IDE, because it kept bugging me, finishing my sentences with fluff that I'd either reject or heavily rewrite. This added some mental overhead that makes it more difficult to focus.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#179

Post by mjburgess »

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to say. Authoring is two-thirds of the point of most communication.

Before this, of course, will be a dramatic "shallowness of thinking" shock that will have to occur before its ill-effects are properly inoculated against. It seems part of the expert aversion to LLMs -- against the credulous lovers of "mediocrity" (cf. https://fly.io/blog/youre-all-nuts/) -- is an early experience of inoculation:

Any "macroscopic usage" of LLMs has, in any of my projects, dramatically impaired my own thinking, stolen decisions-making, and worsened my readiness for necessary adaptions later-on. LLMs are a strictly microscopic fill-in system for me, in anything that matters.

This isn't like calculators: my favourite algorithms for by-hand computation arent being "taken away". This is a system for substituting thinking itself with non-thinking, and radically impairs your readiness (, depth, adaptability, ownership) wherever it is used, on whatever domain you use it on.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#180

Post by smartmic »

> As the educational impact of LLM use only begins to settle with the general population, in this study we demonstrate the pressing matter of a likely decrease in learning skills based on the results of our study.

Fast forward 500 years (about 20 generations), and the dumbing down of the population has advanced so much that films like 'Idiocracy" should no longer be described as science fiction but as reality shows. If anyone can still read history books at that point, the pre-LLM era will seem like an intellectual paradise by comparison.

Post reply on HN