Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automatically Detecting Under-Trained Tokens in Large Language Models

arxiv.org

Post reply on HN
Print view View original

21–27 of 27 posts

Re: Automatically Detecting Under-Trained Tokens in Large Language Models

#21

Post by sebzim4500 »

anewhnaccount3 wrote: post #4

Isn't the solution to just train the tokeniser on the same corpus as the LLM? I'm not sure why reusing tokenisers is so common. Anybody know?

On top of what everyone else has said, even if you are able to train your tokenizer on exactly your training dataset it wouldn't remove all these issues.

The way BPE works you can end up with very rare tokens if they get merged with another token. Imagine you have tokens X and Y, and it happens that almost every X is followed by Y. Then the BPE process would make a new token XY but wouldn't remove the old token which would now be undertrained.

I guess to solve this we'd need to use a more sophisticated merging algorithm than the greedy one.

Re: Automatically Detecting Under-Trained Tokens in Large Language Models

#22

Post by esafak »

There is a random matrix theory derived diagnostic of training that relies on the spectral density of the correlation matrix of the weights. Each layer's spectral density is fit to a truncated power law, and deemed properly trained if the power law exponent alpha is just above two.

https://jmlr.org/beta/papers/v22/20-410.html

Re: Automatically Detecting Under-Trained Tokens in Large Language Models

#23

Post by sdenton4 »

dTal wrote: post #7
bjornsing wrote: post #5

Earlier quoted context omitted.

From the abstract I get the feeling these techniques are useful when you don’t have access to the corpus, as e.g. in the case where you download some open source weights but the corpus is secret. Otherwise I don’t understand why you wouldn’t just compute a histogram over the tokens in (a statistical sample of) the corpus.

> open source weights but the corpus is secret This is oxymoronic; the corpus is the "source". Yet this usage of "open source" is widespread. Maybe we should start calling such models by their rightful name, "freeware".

No, the corpus is not the source. It's data. So we can have concepts of open models, open source, and open data. Any combination of these can be chosen independently.

(Open data and open model but not open source is a bit weird, but not unthinkable: there may be unreleased training tricks or specialized infrastructure such that the source code release is hard or undesirable.)

Re: Automatically Detecting Under-Trained Tokens in Large Language Models

#24

Post by dissuade »

londons_explore wrote: post #12

We shouldn't just be looking for under trained tokens. Tokens are effectively the first layer of the network, but we should also be looking for training data imbalances at every weight at every other layer of the network. When we find them, it might be best to delete weights with hardly any data flowing through them (which might make the model smaller or help generalisation).

We can already compress and/or merge holomorphic models.

Re: Automatically Detecting Under-Trained Tokens in Large Language Models

#25

Post by mycall »

londons_explore wrote: post #12

We shouldn't just be looking for under trained tokens. Tokens are effectively the first layer of the network, but we should also be looking for training data imbalances at every weight at every other layer of the network. When we find them, it might be best to delete weights with hardly any data flowing through them (which might make the model smaller or help generalisation).

> delete weights with hardly any data flowing through them

Isn't that the idea behind sparse networks?

Re: Automatically Detecting Under-Trained Tokens in Large Language Models

#26

Post by dTal »

inbetween wrote: post #18
dTal wrote: post #7

Earlier quoted context omitted.

> open source weights but the corpus is secret This is oxymoronic; the corpus is the "source". Yet this usage of "open source" is widespread. Maybe we should start calling such models by their rightful name, "freeware".

Freeware versus open source is a good point. But freeware typically can't be modified by the recipient, whereas downloadable models and open source code can. So I think there's still a need for a different term, neither open source nor freeware...

I would argue that the kind of modification you can do to a big blob of weights is more akin to fiddling with a binary in a hex editor than modifying source code. It is not the "preferred form" for the source, and you cannot cleanly and easily do things like modify its "alignment" - that is why people speak of "jailbreaking" these models. So I still think "freeware" works as a term.

Re: Automatically Detecting Under-Trained Tokens in Large Language Models

#27

Post by Hugsun »

londons_explore wrote: post #12

We shouldn't just be looking for under trained tokens. Tokens are effectively the first layer of the network, but we should also be looking for training data imbalances at every weight at every other layer of the network. When we find them, it might be best to delete weights with hardly any data flowing through them (which might make the model smaller or help generalisation).

I believe model distillation does this. SparseGPT was a big one, managing to remove 50% of parameters without loosing much accuracy IIRC. I saw a more recent paper citing the SparseGPT one that managed around 70-80% sparsity, pretty impressive stuff.
Post reply on HN