Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: If we train an LLM with “data” instead of “language” tokens

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 52 posts

Ask HN: If we train an LLM with “data” instead of “language” tokens

#1

Post by albert_e »

Is it just traditional ML?

Traditional ML would do a lot of feature extraction and engineering -- from a very specific problem space --before throwing the training compute at it. I think there are good pattern detection, prediction, and anomaly detection models that come out of this.

What happens if we just scrape all data (say metrics like weather, flights, population stats, gps locations, web pages clicks, deep space network observations, and kindergarten grades) and if it were possible to build a model with enough weights for all this diverse data ...

What kind of use case might such a ... Large DATA model ...open up?

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#2

Post by PaulHoule »

Current LLMs seem to struggle with tabular data but there's got to be some answer.

I worked for a startup a while ago that was working on foundation models for tabular data but it was looking at the header and the cell of a csv file to assign some semantics to an individual cell. It competed with more traditional profiling tools that looked at columns as a whole. With that system we'd sometimes treat tabular data using ordinary ML algorithms from scikit-learn.

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#3

Post by h2odragon »

take as an example "bytes" as input, with no further parsing or interpretation given to them. with enough input data and enough time spent iterating on it, you might well come out with decent wordlists, instruction sets, and things like that. if you work the statistics on the wordlists they give you some table based stemmer and thesaurus implications too.

if you go in with a trimmed and parsed dataset its harder to get (what i call) introspection pattern recognition from it because you haven't got enough disparate samples to build a decent universe to compare too. what you assume going in stamps its shape on the whole thing.

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#5

Post by tomohelix »

RLHF is a very important part of the training process. Without it, the LLMs would definitely behave much more similar to a glorified autocomplete than something resembling "intelligence".

So the answer is that if you just feed it a bunch of data, it might be good at spitting out some similarly looking data but ultimately has little use. If you manage to get some experts sitting with it and give it instructions and explain the data, i.e. relating data to concepts, then you may get a model that can generate interesting things based on natural language, because it was taught how to interpret the data with natural language.

That is the next crop of AI I believe. For example, in biotech, people are feeding it protein sequences and teach it what each protein are and how to relate certain concepts with the sequences. The result is an AI that can generate new proteins with certain characteristics and functions.

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#6

Post by mattew »

I’m thinking about data for an LLM from a different angle for ecommerce analytics.

1. Build up a set of instructions on how to interpret a particular type of data in the system prompt

2. Run a set of analysis and instead of outputting the data in tabular form, output natural language versions of the metrics (if they are worth thinking about)

3. Pass the instructions to the LLM to interpret them based on the system prompt

The theory is you can run a number of these analysis and feed the results into another system prompt to do an overall analysis.

It’s a theory at this point but it could work!

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#8

Post by boberoni »

(Disclaimer: I am not an AI expert.) To me, LLMs are a special form "self-supervised learning"[1], which is a special form of unsupervised learning. While supervised learning aims to use input data to predict labels, self-supervised learning uses the context around the input data to predict future data. The key part here is context. This excellent article [2] by Yann Lecun et al. explains how self-supervised learning uses temporal and spatial context to extract patterns, which can then be used to predict future data or generate new data. In fact, prediction and generation go hand-in hand. So, you could definitely apply the techniques of LLMs to arbitrary streams of data with meaningful temporal or spatial patterns.

[1] https://en.wikipedia.org/wiki/Self-supervised_learning

[2] https://ai.meta.com/blog/self-supervised-learning-the-dark-m...

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#9

Post by kylebenzle »

We (I included) have been doing this EXACT thing for a long with with the stock market. We take ALL the data, build a model, then ask for an output buy/sell signal.

The idea for using this type of model, from quants, was applied to language, THAT was the new idea. Now it sounds like you are saying "what if" we took a step backwards, but again, we this is exactly how we use these models now, is trained on data.

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#10

Post by iroddis »

tomohelix wrote: post #5

RLHF is a very important part of the training process. Without it, the LLMs would definitely behave much more similar to a glorified autocomplete than something resembling "intelligence". So the answer is that if you just feed it a bunch of data, it might be good at spitting out some similarly looking data but ultimately has little use. If you manage to get some experts sitting with it and give it instructions and ex…

To save others from searching, RLHF [1] is Reinforcement Learning from Human Feedback

[1] https://en.wikipedia.org/wiki/Reinforcement_learning_from_hu...

Post reply on HN