Can you LLM a custom language?

#1

Post by campervans » Wed, Feb 28, 2024, 9:56 AM UTC

If token limit and accuracy are important, it seems English (or other spoken languages) are no optimal.

They're a butchered product of history and easy verbal noises.

A new custom language seems inevitable, that is concise, unambiguous, rooted in relation with custom words. Replacing common sentences with simple strings such as "Once upon a time..." to "a1"

Most likely alpha-numeric, to minimise tokens, and generate an order of magnitude increase in context window.

Followed by translation back to {language}

Is this possible? Anyone working on it?

(here to be educated)

Re: Can you LLM a custom language?

#2

Post by yorwba » Wed, Feb 28, 2024, 10:31 AM UTC

> Replacing common sentences with simple strings

This is what byte-pair encoding does. It doesn't go quite so far as to allocate only a single token to "Once upon a time", because that string isn't actually that common, but in principle it could.

Trying to get humans to produce content directly in such a concise representation is a waste of time, since LLMs heavily rely on the ability to take whatever content is already available on the internet, which drastically reduces the labor cost of acquiring training data.

Re: Can you LLM a custom language?

#3

Post by campervans » Thu, Feb 29, 2024, 8:03 AM UTC

yorwba wrote: post #2

> Replacing common sentences with simple strings This is what byte-pair encoding does. It doesn't go quite so far as to allocate only a single token to "Once upon a time", because that string isn't actually that common, but in principle it could. Trying to get humans to produce content directly in such a concise representation is a waste of time, since LLMs heavily rely on the ability to take whatever content is alre…

Makes sense, thanks for this