Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLM-Deflate: Extracting LLMs into Datasets

scalarlm.com

Post reply on HN
Print view View original

31–40 of 41 posts

Re: LLM-Deflate: Extracting LLMs into Datasets

#31

Post by gdiamos »

fxj wrote: post #18
moktonar wrote: post #13

Wouldn’t this method be good if applied on humans in job interviews?

how long would it take to do a complete memory dump of your brain by voice stream? days? months? years? this is more like writing one's autobiography.

There are some fun early theoretical ML papers on this topic.

They prove that it is possible to fully clone a brain based on this method.

I think one could theoretically estimate how many queries you would need to make to do it. The worst case is proportional to the number of parameters of the model, i.e. at least 10^15 for a human. At one minute per spoken sample, that comes out to about 2 billion years to clone one human.

I suspect it is not practical without advancements in neural link to increase the bandwidth by billions of times.

I personally like playing around with empirical methods like this blog post to understand the practical efficiency of our learning algorithms like back prop on transformers.

I also try not to invest too much effort into this topic given the ethical issues.

Re: LLM-Deflate: Extracting LLMs into Datasets

#32

Post by gdiamos »

apwell23 wrote: post #5

> This compression is lossy Is compression really lossy? What is an example of lost knowledge?

LLMs have finite entropy (it is related to their training loss) and training typically doesn’t store the residuals.

Some compression methods use LLMs internally and also store the residuals, making them lossless.

Re: LLM-Deflate: Extracting LLMs into Datasets

#33

Post by moktonar »

dragonwriter wrote: post #27
moktonar wrote: post #13

Wouldn’t this method be good if applied on humans in job interviews?

> Wouldn’t this method be good if applied on humans in job interviews? Uhm, no? I mean, some firms do abuse job interviews to pump candidates for usable information, and some have gotten a notable bad reputation for that which impacts their funnel of candidates, but from the article: “Generating comprehensive datasets requires thousands of model calls per topic”—you aren’t going to get a candidate to hang around for…

That is evil, no, I was thinking more about selective knowledge exploration to see if a candidate is fit for the position.

Re: LLM-Deflate: Extracting LLMs into Datasets

#34

Post by moktonar »

fxj wrote: post #18
moktonar wrote: post #13

Wouldn’t this method be good if applied on humans in job interviews?

how long would it take to do a complete memory dump of your brain by voice stream? days? months? years? this is more like writing one's autobiography.

I was thinking about selective knowledge exploration to see if the candidate is fit for the offered position. No need to dump everything

Re: LLM-Deflate: Extracting LLMs into Datasets

#35

Post by gmerc »

omneity wrote: post #7

My gripe with an approach like this is the lack of any grounding to these generated topics. Hallucination accumulates like error in this case so every generation that is conditioned by a previous one (the recursive "hierarchical topic exploration" in TFA). I suspect most of the "leafs" are unusable.

Not different for inference... Just saying.

Re: LLM-Deflate: Extracting LLMs into Datasets

#37

Post by gmerc »

dragonwriter wrote: post #26
imranq wrote: post #8

The claims in this paper don't make sense. There is no proof that anything has been decompressed

“Decompression” is a metaphor, not a fact claim to be proved; it is a description of an approach to generating a dataset from an LLM where most of the potential utility is still fairly explicitly speculative, a jumping off point for further work.

Nope http://arxiv.org/abs/2509.11208

Re: LLM-Deflate: Extracting LLMs into Datasets

#39

Post by estimator7292 »

gdiamos wrote: post #29
omneity wrote: post #7

My gripe with an approach like this is the lack of any grounding to these generated topics. Hallucination accumulates like error in this case so every generation that is conditioned by a previous one (the recursive "hierarchical topic exploration" in TFA). I suspect most of the "leafs" are unusable.

I think it would be interesting to deflate out to a huge dataset and see where this happens. Certainly it will occur as the generated data exceeds the original, eg after 1-10T tokens. I think you could also do this faster by moving down the tree in a depth first manner. Typically I use this for knowledge transfer, style transfer, catastrophic forgetting mitigation, etc and so I don’t go very far. I usually manually r…

Huh. I wonder what good output would look like at extremes. Hallucinations that just happen to be true or something more interesting?

Re: LLM-Deflate: Extracting LLMs into Datasets

#40

Post by ashf023 »

rapatel0 wrote: post #22
fxj wrote: post #16

Earlier quoted context omitted.

The question is: Is it like jpeg compression where the errors do not accumulate but the image comverges to a self inverse compressed image or does the data set converge to a single point which is meaningless?

The transformation function in jpeg (DCT) is generally well defined math. While lossy, most of the information is reprocudable. An LLM is layers and layers of non-linear transformations. It's hard to say exactly how information is accumulated. You can inspect activations from tokens but it's really not clear how to define what the function is exactly doing. Therefore error is poorly understood.

JPEG is similar actually. The DCT is invertible, but the result of the DCT is quantized, which is where some of the compression happens (DCT -> quantization -> IDCT), so the end to end process is not truly invertible. Maybe an analogy to the non-linearities in between the linear steps in deep learning
Post reply on HN