My money is on they used agentic AI coding tools to help set up all the metrics/experiments with traditional models.
Why Large Language Models Fail at Tabular Prediction
31–40 of 42 posts
Re: Why Large Language Models Fail at Tabular Prediction
#32Google's TabFM [1] (and its previous TimesFM) seem the best approach so far on this area
1. https://research.google/blog/introducing-tabfm-a-zero-shot-f...
Re: Why Large Language Models Fail at Tabular Prediction
#33Earlier quoted context omitted.
It has to be 3 in case of a tie. Like the magi system in evangelion.
I don’t know, we saw where that led and I’m not interested in becoming a pool of orange tang yet.
No human souls imprinted in the LLMs (Yet), that I am aware of. I think we're safe for now.
But I'm absolutely joining the Machine Crusade if we have a first Impact event and I survive. Some days I wonder just how flabbergasted Frank Herbert and other pioneers of sci-fi would be at the situation we find ourselves in today.
Re: Why Large Language Models Fail at Tabular Prediction
#34Re: Why Large Language Models Fail at Tabular Prediction
#35Google's TabFM [1] (and its previous TimesFM) seem the best approach so far on this area 1. https://research.google/blog/introducing-tabfm-a-zero-shot-f...
Last time I checked xgboost and lightgbm still outperformed anything NN based for tabular data.
Re: Why Large Language Models Fail at Tabular Prediction
#36Google's TabFM [1] (and its previous TimesFM) seem the best approach so far on this area 1. https://research.google/blog/introducing-tabfm-a-zero-shot-f...
Last time I checked xgboost and lightgbm still outperformed anything NN based for tabular data.
TabPFN [1] is also working in this area and getting better and better. The inference time is still much worse than any of the more traditional models (xg/catboost etc).
Re: Why Large Language Models Fail at Tabular Prediction
#37Earlier quoted context omitted.
I find that an odd take. The paper claims to establish what causes the problem: dimensionality. They are clear in that they don't understand why. But this sort of work is what needs to be done to eventually solve the problem.
Solve what problem? My hammer can't drive screws. Is that a problem to solve?
If you market your hammer as an all purpose tool, yes. If LLM are worse than humans at predicting data it is valid to wonder why, you could think they would be better at it since they are next token predictors.
Re: Why Large Language Models Fail at Tabular Prediction
#38The first thing I'd do if working with an LLM on tabular data is to ask what the best tool would be to work with that data and build up a proper harness to work with the data sensibly. Rawdogging LLM isn't the tool for forecasting like this, as they found.
It’s an interesting question of ‘why not’, though - this was a good read and is upstream of more practical output optimization.
[deleted]
Re: Why Large Language Models Fail at Tabular Prediction
#39Interesting work.
I'm surprised that hypothesis 2 (that CSV serialization format mangles table columns) was falsified. Back in the gpt-3.5-turbo and gpt-4o era, I did needle-haystack tests and found that table format mattered a lot (csv, tsv, markdown). Most models "could not read vertically" for csv (they were horrible), but they could for markdown. I concluded that serialization format or tokenization played a major role.
Nowadays, LLM performance on csvs is much improved (I'm guessing after being explicitly trained on CSV question-answering.) But I still carry the impression that LLMs read columns only by "memorizing" column positions in a format-dependent manner. Maybe this impression is out of date.
Re: Why Large Language Models Fail at Tabular Prediction
#40Watching an LLM interact with the Honeycomb MCP is interesting. When returning a query result it returns not only the tabular data but also an ascii art chart. Presumably they have tested this and it improves the LLM’s ability to interpret the data.