Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which table format do LLMs understand best?

improvingagents.com

Post reply on HN
Print view View original

51–60 of 115 posts

Re: Which table format do LLMs understand best?

#51

Post by mattcollins »

Sharlin wrote: post #5

> where accuracy is paramount > accuracy: 60% Not to mention that the least poorly performing format is probably the stupidest way to encode tabular data, beating even XML. But I guess that’s the new normal because we’re trying to shoehorn conversational AI models to every use case rather than, say, training finetunes that are better at particular tasks. (Yes, of course you can’t train finetunes when the model is a p…

I'm the person who ran the test.

To explain the 60% a bit more...

With small amounts of input data, the accuracy is near 100%. As you increase the size of the input data, the accuracy gradually decreases.

For this test, I intentionally chose an input data set large enough that the LLM would score in the region of 50% accuracy (with variation between formats) in order to maximise the discriminative power of the test.

Re: Which table format do LLMs understand best?

#53

Post by mattcollins »

sega_sai wrote: post #15

Bizarre conclusions when on average all the formats perform poorly with average accuracy of 50%. Sure 60% is better than 40% but they are both unusable if you actually care about numbers...

I'm the person who ran the test.

To hopefully clarify a bit...

I intentionally chose input data large enough that the LLM would be scoring in the region of 50% accuracy in order to maximise the discriminative power of the test.

Re: Which table format do LLMs understand best?

#56

Post by jcheng »

I was curious enough to have Codex create a similar benchmark: https://github.com/jcheng5/table-formats

With 1000 rows and 100 samples and markdown-kv, I got these scores:

- gpt-4.1-nano: 52%

- gpt-4.1-mini: 72%

- gpt-4.1: 93%

- gpt-5: 100%

I was so surprised by gpt-5 getting 100% that I ran it again with 1000 samples. It got 999 correct, and one wrong.

To reproduce it yourself, clone the repo, add a .env file with OPENAI_API_KEY, `uv sync`, and then run:

    uv run inspect eval evals/table_formats_eval.py@table_formats_markdown_kv --model openai/gpt-5 --limit 100
Update: Also, number of rows makes a massive difference, unsurprisingly; at 100 rows, gpt-4.1-nano scores 95%+ for both markdown-kv and csv. Both model and record count seem to matter a lot more than format.

Re: Which table format do LLMs understand best?

#57

Post by simonw »

gpt5 wrote: post #49
Sharlin wrote: post #5

> where accuracy is paramount > accuracy: 60% Not to mention that the least poorly performing format is probably the stupidest way to encode tabular data, beating even XML. But I guess that’s the new normal because we’re trying to shoehorn conversational AI models to every use case rather than, say, training finetunes that are better at particular tasks. (Yes, of course you can’t train finetunes when the model is a p…

Isn't the best performing (markdown tables) and the worst (pipe delimited tables) basically the same format?

The best performing isn't markdown tables, it's markdown key/value pairs:

  ## Record 1
  
  ```
  id: 1
  name: Charlie A0
  age: 56
  city: New York
  department: Operations
  salary: 67896
  years_experience: 7
  project_count: 1
  ```
Which makes sense to me because the problem with formats like CSV and regular markdown tables is that it is too easy for the model to mistakenly associate a value in a row with the wrong header.

Explicit key/value formats like this or YAML or JSON objects make that a lot less likely.

Re: Which table format do LLMs understand best?

#58

Post by jcheng »

jcheng wrote: post #56

I was curious enough to have Codex create a similar benchmark: https://github.com/jcheng5/table-formats With 1000 rows and 100 samples and markdown-kv, I got these scores: - gpt-4.1-nano: 52% - gpt-4.1-mini: 72% - gpt-4.1: 93% - gpt-5: 100% I was so surprised by gpt-5 getting 100% that I ran it again with 1000 samples. It got 999 correct, and one wrong. To reproduce it yourself, clone the repo, add a .env file with O…

gpt-5 also got 100/100 for both CSV and JSON.

    uv run inspect eval evals/table_formats_eval.py@table_formats_csv --model openai/gpt-5 --limit 100
    uv run inspect eval evals/table_formats_eval.py@table_formats_json --model openai/gpt-5 --limit 100

Re: Which table format do LLMs understand best?

#60

Post by Forgeties79 »

crazygringo wrote: post #47
zaidf wrote: post #36

Earlier quoted context omitted.

I've been stunned by how many smart people talk so casually about LLMs becoming better at math. Do they just forget that a calculator that is wrong 1% of the time is a de facto calculator that doesn't work and should not be used?

You realize that when typing into a calculator, you probably hit a wrong key more than 1% of the time? Which is why you always type important calculations twice? I've been stunned by how many smart people talk so casually about how because LLMs aren't perfect, they therefore have no value. Do they just forget that nothing in the world is perfect, and the values of things are measured in degrees?

There’s a big difference between mistyping 1% of the time yourself (human error) and a calculator failing 1% of the time (machine error) and I am willing to bet there isn’t a company out there (maybe a handful of less scrupulous ones) that has knowingly shipped a calculator that got it wrong 1% of the time. Especially in previous decades when countless people were using a dedicated calculator dozens of times a day. Hard to imagine a 1% margin of error was acceptable.

Not to mention now you have the compounded problem of your mistakes plus the calculator’s mistakes.

Post reply on HN