LLM-Deflate: Extracting LLMs into Datasets
scalarlm.com
LLM-Deflate: Extracting LLMs into Datasets
1–10 of 41 posts
Re: LLM-Deflate: Extracting LLMs into Datasets
#2What open source code do you use to pull synthetic data from LLMs?
Re: LLM-Deflate: Extracting LLMs into Datasets
#3Remember those programming books that were like "1000+1 tips for C++", making those with llms would be trivial now.
Re: LLM-Deflate: Extracting LLMs into Datasets
#4I wonder how many cycles of train->extract->train->extract->... you can do before most of your output will be hallucinations.
Re: LLM-Deflate: Extracting LLMs into Datasets
#5> This compression is lossy
Is compression really lossy? What is an example of lost knowledge?
Re: LLM-Deflate: Extracting LLMs into Datasets
#6> This compression is lossy Is compression really lossy? What is an example of lost knowledge?
Think about all the times in llm gets it wrong, the fact that would have helped to get it right is something that was lost. I suppose this isn't proof it's lossy just maybe we don't know how to get the data out.
Or look at it another way LLMs or just text prediction machines, whatever information doesn't help them predict the next token or conflicts with the likelihood of the next token is something that gets dropped.
Or look at it another way these things are often trained on the many terabytes of the internet yet even a 200 billion parameter network is 100 or 200 GB in size. So something is missing, and that is a way better compression ratio then the best known algorithms for lossless compression.
Or we can look at it another way, these things were never built to be lossless compression systems. We can know by looking at how these things are implemented that they don't retain everything they're trained on, they extract a bunch of statistics.
Re: LLM-Deflate: Extracting LLMs into Datasets
#7My gripe with an approach like this is the lack of any grounding to these generated topics. Hallucination accumulates like error in this case so every generation that is conditioned by a previous one (the recursive "hierarchical topic exploration" in TFA).
I suspect most of the "leafs" are unusable.
Re: LLM-Deflate: Extracting LLMs into Datasets
#8The claims in this paper don't make sense. There is no proof that anything has been decompressed
Re: LLM-Deflate: Extracting LLMs into Datasets
#9> This compression is lossy Is compression really lossy? What is an example of lost knowledge?
it's exactly the same as JPEG images being lossy, while you can see image as a whole (and it is enough for 99% of people), you are obviously missing some details
and the more you rely on those details (professional photography, scientific data) the more obvious it is (to the point of image being useless in some cases)
same with LLMs, we are currently testing how far we can go before we seeing obvious issues
Re: LLM-Deflate: Extracting LLMs into Datasets
#10> This compression is lossy Is compression really lossy? What is an example of lost knowledge?
it's exactly the same as JPEG images being lossy, while you can see image as a whole (and it is enough for 99% of people), you are obviously missing some details and the more you rely on those details (professional photography, scientific data) the more obvious it is (to the point of image being useless in some cases) same with LLMs, we are currently testing how far we can go before we seeing obvious issues
whats an example of loss?