Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: If we train an LLM with “data” instead of “language” tokens

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#21

Post by bluecoconut »

I love this question, because this is exactly what I'm currently focused on doing!

I founded approximatelabs to essentially chase this out and show how much rich fruit there is in this space. See approximatelabs.com

It is not traditional ML to me, which has a "row-wise" way of thinking. Instead, we are thinking columnar-ly (like you would if you were an analyst), and representing entities and attributes as tokens (not individual features). To me, describing "data" requires creating tokens that represent the information of the whole of the data. There are a few different ways of approaching this, but technically, the way we are approaching this is representing columns via aggregations (both exact and approximate (sketching algorithms like HLL, TDigest, etc.)). These columns now have representations that can be worked with by transformers in the same way they look at "tokens" of language, now we have "tokens of data" (also think multi-modal patch embeddings with images, but with columns of your data).

To power this, we actively are scraping "all of the data" as you say, and have over 100M tables in our store, and hopefully will push this to over 1B in the next week or so (about to run our next big parse job). We plan to open this dataset to public / research very soon, so looking forward to that. Shortly after that we plan to publish some papers about the actual details of our multi-modal tabular-data foundation models and techniques we have tried and learned from!

Towards the last part of your question: what use case would this model open up? We believe that the entire data stack (currently modern data stack) is about to get a revolution in the way that LLMs caused "symbolic" language models to get revolutionized. Think catalogs that are actually smart and semantic (not just based on the semantics of the metadata, but also of understanding natively the content of the data), observability that is in embedding space and is able to self-explain deviations, and most importantly: accessibility. Since LLMs have proven that they can lower the accessibility barrier to technical concepts, we are expecting and planning on our product being "the new excel", where everyday people can actually leverage the analytic power of things like pandas and sql to answer their questions and trust the results. Heavy bias, but I think the space is about to get very, very hot, and early competitors are starting to pop-up around the chat-bot use-case, and we believe we have a huge edge because we're making our own models + starting at the fundamentals.

Happy to answer any other questions too!

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#22

Post by blovescoffee »

There's a lot to unpack here. First, consider that to the LLM/computer each token is just "data". Second, consider that LLM's are mostly matmul, dot-products, etc., so there has to be some structure to the data and dimensions of the data have to make some sense - but I suppose you could just torch.cat a lot of points (kind of). Anyways, if you want to read about real-world examples of what you're suggesting checkout something like the following foundation model [0] and expand from there.

[0] https://www.earthdata.nasa.gov/news/impact-ibm-hls-foundatio...

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#23

Post by alexander2002 »

bluecoconut wrote: post #21

I love this question, because this is exactly what I'm currently focused on doing! I founded approximatelabs to essentially chase this out and show how much rich fruit there is in this space. See approximatelabs.com It is not traditional ML to me, which has a "row-wise" way of thinking. Instead, we are thinking columnar-ly (like you would if you were an analyst), and representing entities and attributes as tokens (no…

Hi,I am not the best person to hire since i am a new programmer.i would like to intern at your company remotely and see if i can fit some role in your company.

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#24

Post by jameshart »

DanHulton wrote: post #20

...we'd get a model that hallucinates weather forecasts?

All weather forecasts are ‘hallucinations’. When they run ensemble models to predict tropical storm paths there are always outliers that hallucinate a cat 5 going straight through Manhattan or something.

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#25

Post by rdli »

version_five wrote: post #18

Look up autoregressive model. There are already forecasting models that do something like this. I don't know if any use attention, it seems like a pretty obvious thing to have done. Overall this isn't a new idea. Deep learning generally brings a lot less to tabular data because the underlying thing being modelled is much simpler compared to language models that effectively are modeling the human mind + culture so the…

Temporal Fusion Transformer: see https://arxiv.org/abs/1912.09363.

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#26

Post by kylebenzle »

binarymax wrote: post #17
kylebenzle wrote: post #9

We (I included) have been doing this EXACT thing for a long with with the stock market. We take ALL the data, build a model, then ask for an output buy/sell signal. The idea for using this type of model, from quants, was applied to language, THAT was the new idea. Now it sounds like you are saying "what if" we took a step backwards, but again, we this is exactly how we use these models now, is trained on data.

Sorry, but are you suggesting that transformers have been used by quants prior to the language application? LLM these days implies a transformer architecture.

Yes [1]

(1) https://arxiv.org/pdf/2208.08300.pdf

Aug 2022, TRANSFORMER-BASED DEEP LEARNING MODEL FOR STOCK PRICE PREDICTION International Journal of Computational Intelligence and Applications

Something similar was an exercise/project in the NYCDSA "Data Science Boot Camp" I did nearly 2 years ago. I am fairly sure that applying these models to language was secondary, right?

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#27

Post by jmbiven »

Jordan Volz wrote an article speculating about this earlier this year. First time I heard the term Large Data Model, which I love and plan on using.

https://medium.com/@jordan_volz/who-owns-the-future-looking-...

"I won’t claim it’s simple, but as we’ve built AI that can understand human language, we can similarly build AI that understands data...Instead of building a large language model, we instead can build large data models (LDMs?)."

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#28

Post by all2 »

alexander2002 wrote: post #23
bluecoconut wrote: post #21

I love this question, because this is exactly what I'm currently focused on doing! I founded approximatelabs to essentially chase this out and show how much rich fruit there is in this space. See approximatelabs.com It is not traditional ML to me, which has a "row-wise" way of thinking. Instead, we are thinking columnar-ly (like you would if you were an analyst), and representing entities and attributes as tokens (no…

Hi,I am not the best person to hire since i am a new programmer.i would like to intern at your company remotely and see if i can fit some role in your company.

You might leave your email in your profile in case others are interested in contacting you.

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#29

Post by all2 »

kylebenzle wrote: post #9

We (I included) have been doing this EXACT thing for a long with with the stock market. We take ALL the data, build a model, then ask for an output buy/sell signal. The idea for using this type of model, from quants, was applied to language, THAT was the new idea. Now it sounds like you are saying "what if" we took a step backwards, but again, we this is exactly how we use these models now, is trained on data.

One question before I spend 50k training a model: but does it work? IE, does the income yield outpace the cost of training? I'm assuming that the model needs to be retrained at some frequency, though I'm guessing some of that retraining would be model fine tuning... could all of the training be continual fine tuning? Would there be any benefit by training from scratch?

Re: Ask HN: If we train an LLM with “data” instead of “language” tokens

#30

Post by paddy_m »

bluecoconut wrote: post #21

I love this question, because this is exactly what I'm currently focused on doing! I founded approximatelabs to essentially chase this out and show how much rich fruit there is in this space. See approximatelabs.com It is not traditional ML to me, which has a "row-wise" way of thinking. Instead, we are thinking columnar-ly (like you would if you were an analyst), and representing entities and attributes as tokens (no…

Was just looking up data sketches to link to what you all are doing.
Post reply on HN