Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which table format do LLMs understand best?

improvingagents.com

Post reply on HN
Print view View original

81–90 of 115 posts

Re: Which table format do LLMs understand best?

#81

Post by crazygringo »

Forgeties79 wrote: post #60
crazygringo wrote: post #47

Earlier quoted context omitted.

You realize that when typing into a calculator, you probably hit a wrong key more than 1% of the time? Which is why you always type important calculations twice? I've been stunned by how many smart people talk so casually about how because LLMs aren't perfect, they therefore have no value. Do they just forget that nothing in the world is perfect, and the values of things are measured in degrees?

There’s a big difference between mistyping 1% of the time yourself (human error) and a calculator failing 1% of the time (machine error) and I am willing to bet there isn’t a company out there (maybe a handful of less scrupulous ones) that has knowingly shipped a calculator that got it wrong 1% of the time. Especially in previous decades when countless people were using a dedicated calculator dozens of times a day. H…

There isn't a difference in the big picture. Error is error. Even when we have incredibly reliable things, there's error when they interface with humans. Humans have error interfacing with each other.

But you seem to have missed the main point I was making. See? Another error. They're everwhere! ;)

Re: Which table format do LLMs understand best?

#82

Post by xnx »

jcheng wrote: post #79
xnx wrote: post #70

Earlier quoted context omitted.

Cool tool. I tried a few different things to get to work with google/gemini-2.5-pro, but couldn't figure it out.

uv add google-genai uv run scripts/run_benchmarks.py --models google/gemini-2.5-pro --formats markdown_kv --limit 100 And add GOOGLE_API_KEY= to a file called .env in the repo root. Unfortunately I started getting "quota exceeded" almost immediately, but it did give 6/6 correct answers before it crapped out.

Thanks! That worked perfectly.

100 samples:

- gemini-2.5-pro: 100%

- gemini-2.5-flash: 97%

Re: Which table format do LLMs understand best?

#84

Post by __mharrison__ »

mattcollins wrote: post #53
sega_sai wrote: post #15

Bizarre conclusions when on average all the formats perform poorly with average accuracy of 50%. Sure 60% is better than 40% but they are both unusable if you actually care about numbers...

I'm the person who ran the test. To hopefully clarify a bit... I intentionally chose input data large enough that the LLM would be scoring in the region of 50% accuracy in order to maximise the discriminative power of the test.

Can you expand on how you did this?

Re: Which table format do LLMs understand best?

#85

Post by 101008 »

otterley wrote: post #83

Can someone explain why one would want to use an LLM to read tabular data? This is something even trivial code could do while using far fewer compute and energy resources.

Understanding the question is the hard part, that's where the LLM comes as an useful tool.

Re: Which table format do LLMs understand best?

#86

Post by CuriouslyC »

rovr138 wrote: post #74
mattcollins wrote: post #51

Earlier quoted context omitted.

I'm the person who ran the test. To explain the 60% a bit more... With small amounts of input data, the accuracy is near 100%. As you increase the size of the input data, the accuracy gradually decreases. For this test, I intentionally chose an input data set large enough that the LLM would score in the region of 50% accuracy (with variation between formats) in order to maximise the discriminative power of the test.

> As you increase the size of the input data, the accuracy gradually decreases. Interesting. On your section "Limitations and Areas for Further Study", What I'd be curious on future work would be, - changing the order of the data on each table type - changing the order of the questions I'm curious to know if what it fails is the same, if it changes depending on the location, if it's a bias. Is it always a specific qu…

LLMs have documented position biases, with skew towards first and last. This is strongest in messages due to system prompt + current question training data, but it's present in list data in general.

Re: Which table format do LLMs understand best?

#88

Post by beders »

> 60.7%

Why would anyone trust the output of an LLM, if it is barely better than guessing and much much worse than humans?

GPT-5 shows more impressive numbers, but for that particular task, the precision should be 100% - always. No matter how large the data set is or in which format. Why are we doing this?

Re: Which table format do LLMs understand best?

#89

Post by [deleted] »

wagslane wrote: post #77
johnfn wrote: post #68

The article has interesting data. But it’s frustrating to read AI generated text like this: > Performance Optimization: Reducing processing overhead while maintaining accuracy What on earth does it mean that this “optimized performance”? This is nonsensical content. Performance wasn’t even measured, accuracy was. You can tell this was AI generated because “ Reducing processing overhead while maintaining accuracy” wou…

I think they may be referring to token usage, which is mentioned in the article. fewer tokens = higher performance

[deleted]

Re: Which table format do LLMs understand best?

#90

Post by hedgehog »

mitjam wrote: post #87

Was a bit surprised about the low csv performance, in my exp. it‘s very good (use it a lot with Excel and small tables, well below 100 rows). As markdown kv performs so well, I am now curious about TOML.

TOML works decently well both directions, useful if you need structured data out of models or APIs that don't support structured outputs.
Post reply on HN