Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMLingua: Compressing Prompts for Faster Inferencing

github.com

Post reply on HN
Print view View original

21–30 of 51 posts

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#21

Post by smeagull »

TarqDirtyToMe wrote: post #12
behnamoh wrote: post #9

Earlier quoted context omitted.

Came here to mention this. Whenever I hear "alignment" I immediately say "No way am I going to use that shit". Seriously, there's alignment and then there's censorship—the AI creators are using the former when they actually mean the latter. This needs to stop.

My understanding is that in an academic context you’ll hear alignment anytime a model is tuned to accomplish a certain task, not just to steer its political affiliation and idea of ethics I don’t think this models use of alignment implies any sort of censorship, it’s just being tuned to accomplish the task of outputting only important tokens for the target llm

In my experience it means the AI will waste tokens apologizing for it's short comings and ignoring task prompts in favour of it's alignment.

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#24

Post by swyx »

avereveard wrote: post #23

I wonder if this could also be useful in reverse, you'd have a large expensive llm producing a few tokens per sentence about the answer, then a expansion llm forming sentences out of it.

the text to image community has upscalers like this… i wonder if useful

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#26

Post by sroussey »

TarqDirtyToMe wrote: post #2

LLMLingua uses a well-trained small language model after alignment, such as GPT2-small or LLaMA-7B, to detect the unimportant tokens in the prompt and enable inference with the compressed prompt in black-box LLMs, achieving up to 20x compression with minimal performance loss.

What would happen if instead of the long prompt, you just sent the mean of the embeddings of the prompt tokens?

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#27

Post by fabmilo »

This means that we need some new form of preprocessing the data before training LLMs from simple text. Probably just using this compressor and then try to decompress the full text could give some better Supervised Fine Tuned results. Wonder how to deploy this right away. Probably using its own optimized triton inference server?

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#28

Post by cuuupid »

avereveard wrote: post #23

I wonder if this could also be useful in reverse, you'd have a large expensive llm producing a few tokens per sentence about the answer, then a expansion llm forming sentences out of it.

The expansion llm would have to have a pretty good model of language so would likely need to be 7B realm though, but could be useful given we are almost at a time where 7b models can run ubiquitously on most consumer hardware

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#30

Post by passion__desire »

pseudosavant wrote: post #16
mbb70 wrote: post #4

This always seemed like the end game vs. getting a degree in prompt engineering. If you get enough data on "initial prompt attempt" -> "final successful prompt", the whole thing can be replaced by a fine tuned model. You would just select a "prompt rewritter llm" that optimizes for accuracy, cost, alignment etc.

GPT on top of GPT. It is turtles all the way down.

Let's say a particular layperson wants to execute a task. He gives (INPUT OUTPUT) pairs. chatgpt creates a "prompt ( == bytecode)" which captures the essence of those transformations This process is called "Program Fitting" similar to Line fitting or Curve fitting given list of data points. Then this bytecode can then be efficiently run on a smaller distilled CVM (chatgpt virtual machine) diligently chosen by ChatGPT itself since it knows which CVM to best execute the task and then run the (bytecode = prompt) on new similar data. No need to run full ChatGPT. ChatGPT creates its own MoE setups.
Post reply on HN