For it to be a good interjection and not feel like a dumb AI that is just babbling in the background, it would need to have proper timing, and relevant information that includes a model of the minds of people talking. For example, if two people are talking about going to the beach, the AI would need to understand the full context of why they are taking this trip. If the trip is just to enjoy the sun, the AI could have something useful to say on that matter, or if the trip is about surfing, the AI would also factor that in, but the crucial point is that this context is not just listening to keywords or predicting the next token. There would need to be several layers of AI, one layer for predicting subject matter, one layer for predicting intention, another layer for predicting the state of the world that is relevant for that conversation, in addition to the layer of predicting word by word what the response is.
Ask HN: When will LLMs be able to interrupt or interject?
21–30 of 65 posts
Re: Ask HN: When will LLMs be able to interrupt or interject?
#22Earlier quoted context omitted.
I’ve just tried with ChatGPT on iOS, you can press stop then immediately respond while the AI is still generating the response.
I'd prefer a UX where if I start typing, the chatbot immediately stops talking, with a play button which I can use to let it finish predicting whatever it was on about. It's a small tweak, but it would be a significant improvement. Especially when I'm trying to type multi-line input and forget to hit Shift-Enter: there's this immediate context switch to get to the mouse and hit a small button, when what I want to do…
I think this is a great idea, and makes a lot of sense.
It’s a paper cut, but I get annoyed waiting for GPT to finish when I know the output is misdirected.
Ironically - since it’s not a person - hitting “stop” feels too abrasive so I usually wait until the output completes.
Re: Ask HN: When will LLMs be able to interrupt or interject?
#23Check this out. "Negative Latency" :)
https://twitter.com/yoheinakajima/status/1762718034761072653
Re: Ask HN: When will LLMs be able to interrupt or interject?
#24Interjecting requires planning ahead. The way a human interjects is that you have a parallel thought chain going, along with the conversation, as it's happening in real time. In this parallel chain, you are planning ahead. What point am I going to make once we are past this point of conversation? What is the implication of what is being discussed here? (You also are thinking about what the other person is thinking; y…
Take this with a grain of salt because I'm not super well read on llms, but isn't their entire function built on prediction?
Sounds like a reasonable approach could be to have a separate "channel" which focuses entirely on the concept of "where is this conversation going?" could give a pretty good baseline for when and how to interject.
Re: Ask HN: When will LLMs be able to interrupt or interject?
#25“It looks like you're writing a letter. Would you like help?” That didn’t go down so well in the past. https://en.wikipedia.org/wiki/Office_Assistant
True, however in the past we didn’t have LLMs. I would pay good money for a Clippy powered by a capable LLM.
Re: Ask HN: When will LLMs be able to interrupt or interject?
#26“It looks like you're writing a letter. Would you like help?” That didn’t go down so well in the past. https://en.wikipedia.org/wiki/Office_Assistant
True, however in the past we didn’t have LLMs. I would pay good money for a Clippy powered by a capable LLM.
This is an actual sign of the end times. Look it up, and please don't try to resurrect clippy again.
Re: Ask HN: When will LLMs be able to interrupt or interject?
#27One way to do it : After every token inputted by the user (more on that later), you feed it immediately to the LLM which try to predict the next token. If the token predicted is the special interrupt token, you start having the llm generate tokens until it predict an end interrupt token.
It's quite standard nowadays to add some extra special token and then fine-tune a LLM to make it learn how to use it appropriately, by providing a small dataset (1k to 50k) of examples with interruptions (for example "user: Xylophone went to the stadium with Let me stop you right now are you really referring to Xylophone ok thanks for correcting me, it's not Xylophone it's Xander, damn autocorrect!").
llama.cpp has the opposite : an interactive mode where as a human you can interrupt the conversation that the llm is currently generating. But if you interrupt it badly it can make the llm conversation go off-rails.
One problem that result from the usage of tokens is that the user is usually not inputting token but rather characters so you must somehow only process when the characters have stabilized into tokens (for example at word boundaries if your tokeniser has a preprocessing that split on spaces before doing the byte pair encoding). (If you want to process each character on the fly it's getting really tricky because even if at inference you can rewrite the last token in your kv cache, you must somehow create a finetuning dataset to properly learn how to interject based on these partial tokens)
Re: Ask HN: When will LLMs be able to interrupt or interject?
#28I had implemented something like this before, back when GPT-2 was the go-to. It wasn't too complicated. All you need to do is calculate the probability of the AI responding. So in my case, given an input "Joe: Hey how are you, Bob?", calculate the probability that "\nBob:" will come next. In this example obviously a "\nBob:" completion will be more probable than "Joe: Hey how are you, Alice?". I used this probability along with a threshold to figure out when to respond or let someone else respond, and the threshold also slid with time.
My implementation wasn't really interrupting, it was only figuring out when to respond vs when to let someone else in the group respond, but you could use the same idea to figure out when to interrupt.
Re: Ask HN: When will LLMs be able to interrupt or interject?
#29It's probably easier to ask how can you design a text interface that allows people to interrupt, first. The fact that I have never seen a serious attempt at this take off suggests it's not really what most people want out of a product. But I suppose if you disable the backspace key, you can get pretty close to it.
I feel like the ... typing animation lets you interrupt people
Re: Ask HN: When will LLMs be able to interrupt or interject?
#30It’s possible now, no idea why anyone would want this though. The idea is that you want something helpful, and you can do some additional prompting to encourage the model to ask questions but outright derailing the conversation is contrary to what these models are trying to do.
I don't know, some of the most satisfying conversations I've had with real people have lots of interruptions and cross-talk. Anyway, I'd much rather my friend interrupt me than let me prattle on about something stupid.
Good dialogue can be parallel streams of communication; people rarely do strict turn-taking. The half-duplex nature of current chatbots feels very constraining.