Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: When will LLMs be able to interrupt or interject?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Ask HN: When will LLMs be able to interrupt or interject?

#31

Post by jsjohnst »

mlsu wrote: post #11

Interjecting requires planning ahead. The way a human interjects is that you have a parallel thought chain going, along with the conversation, as it's happening in real time. In this parallel chain, you are planning ahead. What point am I going to make once we are past this point of conversation? What is the implication of what is being discussed here? (You also are thinking about what the other person is thinking; y…

> LLM does not have any of this, architecturally, it just has the text itself.

I feel like you are maybe being a bit too focused on specifics of how the LLM works where as:

> The way a human interjects is that you have a parallel thought chain going

You are more abstract in the human case.

They really don’t need to be different here. The LLM could be running predictions in parallel each time you type another token playing out where the conversation is going. You could then layer on another model which blends these together (vaguely like MoE works) and is trained on opportune times to interject. Think of it like a chess playing AI, but rather with the goal of interjecting appropriately vs Checkmate.

The amount of compute power to run all these inferences at once would be fairly expensive, but it’s technically all possible today and wouldn’t be that much different than the human case for this specific scenario imho.

Re: Ask HN: When will LLMs be able to interrupt or interject?

#32

Post by Andrex »

b112 wrote: post #26
chorsestudios wrote: post #25

Earlier quoted context omitted.

True, however in the past we didn’t have LLMs. I would pay good money for a Clippy powered by a capable LLM.

This is an actual sign of the end times. Look it up, and please don't try to resurrect clippy again.

> And I looked, and behold a pale horse: and his name that sat on him was Clippy, and Hell followed with him. And power was given unto them over the fourth part of the earth, to kill with sword, and with hunger, and with death, and with the beasts of the earth.

> Revelation 6:8 (parody, please @God don't convict me under the "don't EVER rewrite ANY of Revelation, specifically" clause in Chapter 22, please!)

Re: Ask HN: When will LLMs be able to interrupt or interject?

#33

Post by vrighter »

GistNoesis wrote: post #27

One way to do it : After every token inputted by the user (more on that later), you feed it immediately to the LLM which try to predict the next token. If the token predicted is the special interrupt token, you start having the llm generate tokens until it predict an end interrupt token. It's quite standard nowadays to add some extra special token and then fine-tune a LLM to make it learn how to use it appropriately,…

of course it needs to work faster than the user can type, and the gpu would be screaming the entire time.

Re: Ask HN: When will LLMs be able to interrupt or interject?

#34

Post by nicklecompte »

"Take control of the conversation"...and do what? Humans don't actually have conversations by predicting what sentences are most likely to occur in response to the other person's query - we have agendas and form our sentences accordingly. So if we interrupt another person speaking, it's because we have a specific, often personal reason to do so: perhaps we want to steer the topic of conversation to something we are interested in; perhaps somebody is suggesting a clearly bad idea; we might correct misinformation; take irrational personal offense; and so on. Interruptions are by design antagonistic, and among humans they involve a conflict of agendas.

But LLMs don't have any agenda whatsoever - they are not capable of having goals or motivations. So why are they interrupting? Are they reading your mind and understanding your goals before you even finish typing them? It's hard to see an LLM having a coherent way to interrupt based purely on a probabilistic view of language.

It would be very annoying if a human constantly interrupted you because they were "aligned with your agenda" and thought they were being helpful. LLMs would probably be much worse, even if they were able to reliably infer what you wanted. For an LLM to be useful, you kind of have to coax it along and filter out a lot of empty verbiage - it seems downright counterproductive to have that verbiage blasted at you by a chatbot that interrupts your typing.

I could see LLMs interrupting if you are typing something clearly false or against TOS. But that would require an LLM which reliably understands things are clearly false or against TOS and hence requires a solution to jailbreaking....so in 2024 I think it would just be an incredibly annoying chatbot. In general I think any interruption behavior would be artificially programmed to make the LLM seem "realistic," and it won't work.

Re: Ask HN: When will LLMs be able to interrupt or interject?

#35

Post by Terretta »

As of this comment, sibling comments speculate this should be possible, or say they've seen a demo, or etc. Maybe they're talking about audio?

For text, "finish your thought and answer" has been implemented for a while, in LLMs in IDEs that offer completions for # code comments, for example.

One of the faster implementations is in the new Zed editor. Open the Assistant pane with your OpenAI GPT-4 key, and once you're into the conversation, it will offer auto-completions of your own prompt to it, before you submit.

Often these autocompletes finish the question and then contain the answer, like an impatient listener mentally finishing your sentence so they can say what they think. This is without having submitted the question to the chat interface.

Note that as Zed has implemented this, the realtime "finish your thought for you" mode is a dumber faster model, but as your context builds, it interrupts right more often.

You can also start your next prompt while it's unspooling the last one.

Re: Ask HN: When will LLMs be able to interrupt or interject?

#36

Post by guizzy »

GistNoesis wrote: post #27

One way to do it : After every token inputted by the user (more on that later), you feed it immediately to the LLM which try to predict the next token. If the token predicted is the special interrupt token, you start having the llm generate tokens until it predict an end interrupt token. It's quite standard nowadays to add some extra special token and then fine-tune a LLM to make it learn how to use it appropriately,…

Great idea, and this is kind of solution is why improving performance of smaller local models is important, not just the highest quality state-of-the-art (local or cloud) models.

Re: Ask HN: When will LLMs be able to interrupt or interject?

#38

Post by layer8 »

chorsestudios wrote: post #25
layer8 wrote: post #18

“It looks like you're writing a letter. Would you like help?” That didn’t go down so well in the past. https://en.wikipedia.org/wiki/Office_Assistant

True, however in the past we didn’t have LLMs. I would pay good money for a Clippy powered by a capable LLM.

People generally don’t like being interrupted by “smart” comments, be it by an LLM or by other people. It requires tremendous contextual and situational awareness to know when it is appropriate.

Re: Ask HN: When will LLMs be able to interrupt or interject?

#39

Post by brookst »

nicklecompte wrote: post #34

"Take control of the conversation"...and do what? Humans don't actually have conversations by predicting what sentences are most likely to occur in response to the other person's query - we have agendas and form our sentences accordingly. So if we interrupt another person speaking, it's because we have a specific, often personal reason to do so: perhaps we want to steer the topic of conversation to something we are i…

Human conversations are often multithreaded. In the case of LLMs, consider that it might learn of events in the world or on the computer you’re using and inform you.

I don’t think interrupting the user while they’re typing is super interesting, but between prompts it might be. “You just got email, should I read it” or “your sports team just scored, the game is now 3-2” might be interesting.

Re: Ask HN: When will LLMs be able to interrupt or interject?

#40

Post by jahsome »

Andrex wrote: post #32
b112 wrote: post #26

Earlier quoted context omitted.

This is an actual sign of the end times. Look it up, and please don't try to resurrect clippy again.

> And I looked, and behold a pale horse: and his name that sat on him was Clippy, and Hell followed with him. And power was given unto them over the fourth part of the earth, to kill with sword, and with hunger, and with death, and with the beasts of the earth. > Revelation 6:8 (parody, please @God don't convict me under the "don't EVER rewrite ANY of Revelation, specifically" clause in Chapter 22, please!)

A little "light blasphemy" never hurt anyone!
Post reply on HN