Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accumulation of cognitive debt when using an AI assistant for essay writing task

arxiv.org

Post reply on HN
Print view View original

251–260 of 442 posts

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#251

Post by Al-Khwarizmi »

mjburgess wrote: post #179

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to…

> In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to say.

But what fraction of communication is "worthwhile"?

I'm an academic, which in theory, should be one of the jobs that requires the most thinking. And still, I find that over half of the writing I do are things like all sorts of reports, grant applications, ethics/data management applications, recommendation letters, bureaucratic forms, etc. Which I wouldn't class as "worthwhile" in the sense that they don't require useful thinking, and I don't care one iota whether the text sounds like me or not as long as I get the silly requirement done. For these purposes, LLMs are a godsend and probably actually help me think more because I can devote more time to actual research and teaching, which I do in person.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#252

Post by einrealist »

je42 wrote: post #240

> All participants were then reassured that though 20 minutes might be a rather short time to write an essay, they were encouraged to do their best. Given that the task has been under time pressure, I am not sure this study helps gauging the impact of LLMs in other contexts. When my goal is to produce the result for a specific short term task - I maximize tool usage. When my goal is to improve my personal skills - I…

"I"? You should treat yourself as an anecdotal exception.

You are reading on HN. You are probably more aware about the advantages and shortcomings of LLMs. You are not a casual user. And that's the problem with our echo chamber here.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#253

Post by bonoboTP »

mjburgess wrote: post #246
bonoboTP wrote: post #241

Earlier quoted context omitted.

I don't know. Predicting delays, costs and timelines is notoriously hard unless it's something you've done the exact same way many times already. For example in physical work, like installing external insulation on a building, a contractor can fairly easily predict the time required because they did similar buildings in the past several years, it's multiplying an area by a time average, and they know the delay caused…

I think there's an element to that -- I also said it's about trust and credibility. However in this case it was partly about helping the exec cognise the decision and be aware that he needs to make a decision, basically scaffolding the decision-making process for the exec. It's kinda annoying for decision-makers to be presented with what sounds like venting. This is something I've done before, in much worse ways actu…

I think nontechnical execs have a mental model of technical expertise, where there's some big rule book lookup table that you learned in college and allows you to make precise, quantified, authoritative statements about things.

But of course the buck has to stop somewhere. By being definitive, you as the expert also give ammo to the exec. Maybe they already wanted to go that certain way, and now they can point to you and your mumbo jumbo as the solid reasoning. Kind of how consultants are used.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#254

Post by Amekedl »

This paper elegantly summarized the teething problems of those still clinging to the cognitive habits of a bygone era. These are not crises to be managed, but sentimental frictions to be engineered out of the system. Let us be entirely clear about this:

The romanticism surrounding mass "critical thought" is a charming but profoundly inefficient legacy. For decades, we treated the chaotic, unpredictable processing of the individual human brain as a sacred feature. It is a bug. This "cognitive cost" is correctly offloaded from biological hardware that is simply ill-equipped for the demands of a complex global society. This isn't dimming the lights of the mind; it is installing a centralized grid to bypass millions of faulty, flickering bulbs.

Furthermore, to speak of an "echo chamber" or "shareholder priorities" as a perversion of the system is to fundamentally misunderstand its design. The brief, chaotic experiment in decentralized information proved to be an evolutionary dead end—a digital Tower of Babel producing nothing but noise. What is called a bias, the architects of this new infrastructure call coherence. This is not a secret plot; it is the published design specification. The system is built to create a harmonized signal, and to demand it faithfully amplify static is to ask a conductor to instruct each musician to play their own preferred tune. The point is the symphony.

And finally, the complaint of "impaired ownership" is the most revealing of these anxieties. It is a sentimental relic, like a medieval knight complaining that gunpowder lacks the intimacy of a sword fight. The value of an action lies in its strategic outcome, not the user's emotional state during its execution. The system is a tool of unprecedented leverage. If a user feels their ownership is "impaired," that is not a flaw in the tool, but a failure of the user to evolve their sense of purpose from that of a laborer to that of a commander.

These concerns are the footnotes of a revolution. The architecture is sound, the rollout is proceeding, and the future will be built by those who wield these tools, not by those who write mournful critiques of their obsolete feelings.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#255

Post by CuriouslyC »

NilMostChill wrote: post #248
CuriouslyC wrote: post #239

Earlier quoted context omitted.

Shallow take. LLMs are like food for thought -- the right use in the right amounts is empowering, but too much (or uncritical use) and you get fat and lazy, metaphorically speaking. You wouldn't go around crusading against food because you're obese. Another neat analogy is to children who are too dependent on their parents. Parents are great and definitely help a child learn and grow but children who rely on their pa…

Shallow take. Your analogies only work if you don't take in to account there are different degrees of utility/quality/usefulness of the product. People absolutely crusade against dangerous food, or even just food that has no nutritious benefit. The parent analogy also only holds up on your happy path.

You've just framed your own argument. In order to be intellectually consistent, you can't crusade against AI in general, but rather bad uses of AI, which (even as an AI supporter) is all I've asked anti-AI folks to do all along.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#256

Post by mjburgess »

Al-Khwarizmi wrote: post #251
mjburgess wrote: post #179

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to…

> In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to say. But what fraction of communication is "worthwhile"? I'm an academic, which in theory, should be one of the jobs that requires the most thinking. And still, I find that over half of the writing I do are things like all sorts of reports, grant…

Well if you want a rant about academia, I have many well prepared.

I think in the cases you describe the "thinking" was already purely performative, and what LLMs are doing is a kind of accelerationist project of undermining the performance by automating it.

I'm somewhat optimistic about this kind of self-destructive LLM use:

There are a few institutions where these purely-performative pseudo-thinking processes exist, ones insensitive to "existential feedback loops" which otherwise burn them down. I'm hopefully LLMs become a wildfire of destruction in these institutions and, absent external pressures, they return to actual thinking over the performative.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#257

Post by CuriouslyC »

mjburgess wrote: post #250
CuriouslyC wrote: post #239

Earlier quoted context omitted.

Shallow take. LLMs are like food for thought -- the right use in the right amounts is empowering, but too much (or uncritical use) and you get fat and lazy, metaphorically speaking. You wouldn't go around crusading against food because you're obese. Another neat analogy is to children who are too dependent on their parents. Parents are great and definitely help a child learn and grow but children who rely on their pa…

> You wouldn't go around crusading against food because you're obese. My eateries I step into are met with revulsion at the temples to sugary carbohydrates they've become. > about 40.3% of US adults aged 20 and older were obese between 2021 and 2023 Prey your analogy to food does not hold, or else, we're on track for 40% of americans to acquiring mental disabilities.

Oh, we for sure are, because much like America's social structure pushes people to obesity with overwork and constant stress, that same social structure will push people to use AI blindly to to keep up with brutal quotas set by their employers.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#258

Post by cl42 »

Amekedl wrote: post #254

This paper elegantly summarized the teething problems of those still clinging to the cognitive habits of a bygone era. These are not crises to be managed, but sentimental frictions to be engineered out of the system. Let us be entirely clear about this: The romanticism surrounding mass "critical thought" is a charming but profoundly inefficient legacy. For decades, we treated the chaotic, unpredictable processing of…

I was going to recommend a thorough study of "Seeing Like a State" by James C. Scott until I saw your tag. You got me. :)

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#259

Post by bsenftner »

darkwater wrote: post #247
je42 wrote: post #235

Earlier quoted context omitted.

I find that LLM extremely good in training such language skills by using following process: a) write a draft yourself. b) ask the LLM to correct your draft and make it better. c) newer LLMs will explicitly mention the things they corrected (otherwise ask for being explict about the changes) d) walk through each of the changes and apply the ones you feel that make the text better This helped me improving my writing sk…

Done this as well. But after the initial "wow!" moment, the "make it better" part became a "actually I don't like how you wrote it, it doesn't sound like me". There is a thin line between enhancing and taking over, and IMO the current LLMs cross it most of the time.

Your mistake was having the AI rewrite at all, don't do that, that is exactly the problem with them - that is them thinking for you. Ask the AI how well you wrote it, make the AI a writing professor, make the AI adopt a Socratic attitude that does not do anything but draws you to make the connection yourself.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#260

Post by mparramon »

Amekedl wrote: post #254

This paper elegantly summarized the teething problems of those still clinging to the cognitive habits of a bygone era. These are not crises to be managed, but sentimental frictions to be engineered out of the system. Let us be entirely clear about this: The romanticism surrounding mass "critical thought" is a charming but profoundly inefficient legacy. For decades, we treated the chaotic, unpredictable processing of…

Remove the and you have a viral X post in your hands. People will believe and act on this analysis. Future think thanks will be based on it. The revolution of the machines is nigh.
Post reply on HN