Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMLingua: Compressing Prompts for Faster Inferencing

github.com

Post reply on HN
Print view View original

31–40 of 51 posts

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#31

Post by nathan_compton »

behnamoh wrote: post #9
TarqDirtyToMe wrote: post #2

LLMLingua uses a well-trained small language model after alignment, such as GPT2-small or LLaMA-7B, to detect the unimportant tokens in the prompt and enable inference with the compressed prompt in black-box LLMs, achieving up to 20x compression with minimal performance loss.

Came here to mention this. Whenever I hear "alignment" I immediately say "No way am I going to use that shit". Seriously, there's alignment and then there's censorship—the AI creators are using the former when they actually mean the latter. This needs to stop.

It amazes me that this amazing new technology comes out and there is a group of people who are like "NO, NOT IF IT CAN'T TELL RACIST JOKES!"

I agree that like "tone" alignment is silly and pointless for models in the public domain, but if I were a big company who wanted to keep customers I'd align my models this way. It isn't censorship, its marketing.

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#34

Post by PoignardAzur »

avereveard wrote: post #23

I wonder if this could also be useful in reverse, you'd have a large expensive llm producing a few tokens per sentence about the answer, then a expansion llm forming sentences out of it.

Some teams have researched ways to do this.

For instance, you can have a smaller model generate ten tokens in sequence, and then ask the larger mode "given these N tokens, what is the token N+1" ten times in parallel.

If the large and small model agree on, say, the first 7 tokens, then you keep these and throw the next 3 away and start over. So you still have to run the large model for each token, but you can at least do batch calculations (which is a lot more efficient, because loading layer weights is the bottleneck, not matrix ops).

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#36

Post by samus »

passion__desire wrote: post #30
pseudosavant wrote: post #16

Earlier quoted context omitted.

GPT on top of GPT. It is turtles all the way down.

Let's say a particular layperson wants to execute a task. He gives (INPUT OUTPUT) pairs. chatgpt creates a "prompt ( == bytecode)" which captures the essence of those transformations This process is called "Program Fitting" similar to Line fitting or Curve fitting given list of data points. Then this bytecode can then be efficiently run on a smaller distilled CVM (chatgpt virtual machine) diligently chosen by ChatGPT…

For all we know, ChatGPT 4 might function like that

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#39

Post by intellectronica »

dmos62 wrote: post #37

Excuse the hijack: what are the most powerful language models one can run on any smartphone (without meaningful delay)?

Phi-2 ( https://www.microsoft.com/en-us/research/blog/phi-2-the-surp... ) may be the best right now. Gemini Nano by Google DeepMind is a close second.

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#40

Post by samus »

joelthelion wrote: post #22

I wonder if, as humans, we could benefit from this. Could we learn to read this compressed lingo?

The model doing the compression is trained with a human language corpus. Also, this is a generic procedure to feed another model trained on a similar corpus. Therefore, I'd not expect the compressing model to do anything exotic.

Btw., humans are quite good at compressing as well. SMS used to be billed per 128 characters. Also, any slang or technical jargon are attempts at compression. These are how people push the limits of expressivness and contribute to language evolution.

Post reply on HN