Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accumulation of cognitive debt when using an AI assistant for essay writing task

arxiv.org

Post reply on HN
Print view View original

191–200 of 442 posts

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#191

Post by jrflowers »

mjburgess wrote: post #179

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to…

> This is a system for substituting thinking itself with non-thinking

One of my favorite developments on the internet in the past few years is the rise of the “I don’t think/won’t think/didn’t think” brag posts

On its own it would be a funny internet culture phenomenon but paired with the fact that you can’t confidently assume that anybody even wrote what you’re reading it is hilarious

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#192

Post by Ekaros »

mjburgess wrote: post #179

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to…

> I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people

Now one like me might go and ask how much of communication is actually worthwhile? Sometimes I consider that there is lot of communication that might not actually be. It is still done, but if no one actually reads it, why not automate generation.

Not to say there is not significant amount of stuff you actually want to get right.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#193

Post by hcta »

> We recruited a total of 54 participants for Sessions 1, 2, 3, and 18 participants among them completed session 4.

> We used electroencephalography (EEG) to record participants' brain activity in order to assess their cognitive engagement and cognitive load

> We performed scoring with the help from the human teachers and an AI judge (a specially built AI agent)

Next up: your brain on psych studies

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#194

Post by mijoharas »

jrflowers wrote: post #191
mjburgess wrote: post #179

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to…

> This is a system for substituting thinking itself with non-thinking One of my favorite developments on the internet in the past few years is the rise of the “I don’t think/won’t think/didn’t think” brag posts On its own it would be a funny internet culture phenomenon but paired with the fact that you can’t confidently assume that anybody even wrote what you’re reading it is hilarious

> One of my favorite developments on the internet in the past few years is the rise of the “I don’t think/won’t think/didn’t think” brag posts

Sorry, I can't immediately think of what you're talking about. Could you link to an example so I can get a feel for it?

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#195

Post by nottorp »

alex77456 wrote: post #145
nottorp wrote: post #98

Earlier quoted context omitted.

> Cycling has been shown time & again to be a great way to increase fitness. Compared to sitting on your butt in a car or public transport. Perhaps not compared to walking everywhere and chasing the antelope you want to cook for lunch. I think what he meant is that both bicycles and LLMs are a force multiplier and you still provide the core of the work, but not all of the work any more.

Cycling, in my experience, is usually way more intense than walking or even running/jogging. It just lets you cover larger distance and gives you more control over how your energy is used. With the example of LLMs, sure, you could cycle the initial destination you were meant to walk to - write an article with its help, save a few hours and call it a day. Or you could cycle further and use the saved time to work on so…

If we're nitpicking, are we talking about cycling as a sport or cycling as a means for getting from point A to point B?

I'm sure cultures where they cycle to everywhere all the time take it easier than cultures where going out for a bike ride is an event.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#196

Post by panstromek »

jonplackett wrote: post #182

I think we need to shift our idea of what LLMs do and stop thinking they are ‘thinking’ in any human way. The best mental description I have come up with is they are “Concept Processors”. Which is still awesome. Computers couldn’t understand concepts before. And now they can, and they can process and transform them in really interesting and amazing ways. You can transform the concept of ‘a website that does X’ into c…

Concept Processor actually sounds pretty good, I like it. That's pretty close to how I treat LLMs.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#197

Post by mjburgess »

sesm wrote: post #183
mjburgess wrote: post #179

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to…

One of the effects on software development is: the fact that you submitted a PR with any LoC count doesn't mean that you did any work. You need to explain your solution and answer questions to prove that.

The next stage of this issue is: how do you explain something you didn't write?

The LLM-optimist view at the moment, which takes on board the need to review LLMs, assumes that this review capability will exist. I cannot review LLM output on areas outside of my expertise. I cannot develop the expertise I need if I use an LLM in-the-large.

I first encountered this issue ~year-ago when using an LLM to prototype a programming language compiler (a field I knew quite well anyway) -- but realised that very large decisions about the language were being forced by LLM implementation.

Then, over the last three weeks, I've had to refresh my expertise in some areas of statistics and realised much of my note taking with LLMs has completely undermined this process -- the effective actions have been, in follow on, traditional methods: reading books, watching lectures, taking notes. The LLM is only a small time saver, "in the small" once I'm an expert. It's absolutely disabling as a route back to expertise.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#198

Post by jstummbillig »

mjburgess wrote: post #179

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to…

I see it as more of a calibration, revolving around understanding what an AI is inherently not able to do – decide what YOU want – and stopping to be weird about that. If you chose to stop being involved in a process and mold it, then your relationship to that process and the outcome will necessarily change. Why would we be surprised by that?

As soon as we stop treating AI like mind readers things will level out.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#199

Post by kibibu »

Ekaros wrote: post #192
mjburgess wrote: post #179

It feels, more and more, that LLMs will be another technology that society will inoculate itself against. It's already starting to happen in education: teachers conversing with students, observing them learn, observing them demonstrate their skills. In business, quickly I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people -- as authors of what they want to…

> I'd say, we will realise that the vast majority of worthwhile communication necessarily must be produced by people Now one like me might go and ask how much of communication is actually worthwhile? Sometimes I consider that there is lot of communication that might not actually be. It is still done, but if no one actually reads it, why not automate generation. Not to say there is not significant amount of stuff you…

> It is still done, but if no one actually reads it, why not automate generation.

There's a reason the real-estate industry has been able to go all-in on using AI to write property listings with almost no consumer kickback (except when those listings include hallucinated schools).

We're already used to treating them with skepticism, and nobody takes them at face value.

Re: Accumulation of cognitive debt when using an AI assistant for essay writing task

#200

Post by fhd2 »

tuatoru wrote: post #177
fhd2 wrote: post #171

Earlier quoted context omitted.

Appears to align with good old Ironies of Automation [1]. If humans just review and rubber stamp results, they do a pretty terrible job at it. I've been thinking for a while now that in order to truly make augmented workflows work, the mode of engagement is central. Reviewing LLM code? Bah. Having an LLM watch over my changes and give feedback? Different story. It's probably gonna be difficult and not particularly po…

Didn't realise the pedigree of the idea went back to 1983. I read about this in a book "Our Robots, Ourselves". That talked about airline pilots' experience with auto-land systems introduced in the late 1990s/ early 2000s. As you'd expect after having read Ironies of Automation, after a few near misses and not misses, auto-land is not used any more. Instead, pilot augmentation with head-up displays is used. What is t…

Certainly a relatively tight feedback loop, but not too tight. Syntax errors are very tight, but non negotiable: Fix it now.

Test failures are more explicit, you run tests when you want to and deal with the results.

Code review often has a horrible feedback loop - often days after you last thought about it. I think LLMs can help tighten this. But it can't be clippy, it can't interrupt you with things that _may_ be problems. You have to be able to stay in the flow.

For most things that make programmers faster, I think deterministic tooling is absolutely key, so you can trust it rather blindly. I think LLMs _can_ be really helpful for helping you understand what you changed and why, and what you may have missed.

Just some random ideas. LLMs are amazing. Incorporating them well is amazingly difficult. What tooling we have now (agentic and all that) feels like early tech demos to me.

Post reply on HN