Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMLingua: Compressing Prompts for Faster Inferencing

github.com

Post reply on HN
Print view View original

41–50 of 51 posts

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#41

Post by TarqDirtyToMe »

smeagull wrote: post #21
TarqDirtyToMe wrote: post #12

Earlier quoted context omitted.

My understanding is that in an academic context you’ll hear alignment anytime a model is tuned to accomplish a certain task, not just to steer its political affiliation and idea of ethics I don’t think this models use of alignment implies any sort of censorship, it’s just being tuned to accomplish the task of outputting only important tokens for the target llm

In my experience it means the AI will waste tokens apologizing for it's short comings and ignoring task prompts in favour of it's alignment.

This does not seem relevant to the alignment discussed in the paper. It seems to be explicitly out of scope:

> The potential harmful, false or biased responses using the compressed prompts would likely be unchanged. Thus using LLMLingua has no inherent benefits or risks when it comes to those types of responsible AI issues.

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#42

Post by MrYellowP »

I was working on the same thing months ago and it works, but it was a purely trial and error way of doing it and the compressed prompts, naturally, wouldn't necessarily work for different LLMs easily.

I am not actually convinced this is a good idea, though. This path eventually leads to a "prompt compiler" that compiles prompts into byte code for a future "more efficient" LLM to understand.

Oh and it definitely didn't require its own language model. All it required was finding how many letters one can remove from a word and which words can be completely omitted.

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#43

Post by nextaccountic »

TarqDirtyToMe wrote: post #13
thebeardisred wrote: post #5

Wild. if I'm reading this correctly it's effectively a sort of "zip" algorithm for both the inputs and outputs of a prompt based model. thus, it allows a user to compress their request down to the minimal token size which retains the same semantics. In effect, this then allows a user to encode a more dense set of tokens into the original request. Does that sound about right?

Sounds right to me. I think it’s fun that is this may be the only compression algorithm where the output is still human understandable. It reads like a slightly garbled version of what someone writing down bullet point notes of a lecture might write. It’s so rare that the human optimized and machine optimized versions of an input are so similar

Is there a text file with many input/output pairs? I couldn't find it in the readme

The examples folder contain jupyter notebooks, there's also some videos and papers, while I just want to see an example text compressed

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#44

Post by refulgentis »

3abiton wrote: post #8
refulgentis wrote: post #7

Earlier quoted context omitted.

Yes you're correct -- it's a really interesting thing, in that it reminds me of early 2023 when people would "compress" prompts by having ChatGPT rewrite it itself into something smaller. There's really no substantive difference between that and what they're doing here, other than they're purposefully using a crappier model than GPT 3.5/ChatGPT to increase the cost savings. For example, the first set of graphics is d…

There is a need for a comparison, otherwise I find your assessment of the performance a "bit" subjective.

Please, by all means! I didn't mean to imply I have data or that you need to accept my comment as a scientific data-backed conclusion. :) I just have the lived experience of ~0 ML models performing better at 0-shot than 5-shot. That would be a good sign of AGI, in fact, now that I think about it...the model being able to workaround good instructions with bad examples.

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#45

Post by Kevin09210 »

samus wrote: post #35
baq wrote: post #25

Redundancy is resiliency - wonder if there’s still enough error correction in the compressed language?

Error connection is not really required here since there is no lossy communication medium.

Until your prompts looks like the following:

with the following text:

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#46

Post by TarqDirtyToMe »

nextaccountic wrote: post #43
TarqDirtyToMe wrote: post #13

Earlier quoted context omitted.

Sounds right to me. I think it’s fun that is this may be the only compression algorithm where the output is still human understandable. It reads like a slightly garbled version of what someone writing down bullet point notes of a lecture might write. It’s so rare that the human optimized and machine optimized versions of an input are so similar

Is there a text file with many input/output pairs? I couldn't find it in the readme The examples folder contain jupyter notebooks, there's also some videos and papers, while I just want to see an example text compressed

There’s some examples on the website: https://llmlingua.com/

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#47

Post by iofu728 »

thebeardisred wrote: post #5

Wild. if I'm reading this correctly it's effectively a sort of "zip" algorithm for both the inputs and outputs of a prompt based model. thus, it allows a user to compress their request down to the minimal token size which retains the same semantics. In effect, this then allows a user to encode a more dense set of tokens into the original request. Does that sound about right?

In fact, it can be seen as semantic communication, which is defined by Shannon.

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#48

Post by cyanydeez »

arthurcolle wrote: post #20
cyanydeez wrote: post #3

Earlier quoted context omitted.

“Why waste time say lot word when few word do trick” -Kevin Malone

Perfection. Key insight. "Few Word [is] All Need" (with a robust enough foundation model) Linked for the culture: https://www.youtube.com/watch?v=bctjSvn-OC8&t=4s Sleep big last night

"Kevin, are you saying 'See the World' or Sea World?" -- Jim

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#49

Post by wklm »

I copied all the the text from this thread and compressed it, the result is:

``` {'compressed_prompt': '\t | submit\twout\nLLMLing byqtyTo\n\n. ". which\nq1 that only down human\nnextaccount many examples,pressed\nq4\n\n as semantic\n" having noreings of isating withoutre this and\n31] a\n\n the0 of, to workaroundqTo\ning after and in loss\n\n time say -. Word a\nb-leep big\n\nsr the\namshipIqToMy hear alignment to its andics this only target\n will tokensq be: The the usinging\n\nbeamIt" mying\na large expensive am\n\n generate larger"3 loading).\n\nThe expansionB run has this if\nhas] agents think it\n\npyinstall into game in " promptter\nos\n particular (. ( == transformations to given smaller) ownups\n\n this better [] thewithout\n\n. is -Error medium\n\nChat GPT4 doesn't know what to do with it: https://chat.openai.com/share/73bc7b96-4453-4a6e-944d-d9d4c5...

Re: LLMLingua: Compressing Prompts for Faster Inferencing

#50

Post by the_omegist »

Made me think of Speedtalk by Heinlein [0].

One way to increase the context window, I thought, would be to teach the LLM a compressed language based on abbreviations etc and to have some compressing/uncompressing script do the translating with the LLM. That would allow longer prompts too.

Not as sophisticated as this LLMLingua but good enough for basic users.

[0] https://en.wikipedia.org/wiki/Speedtalk

Post reply on HN