When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
1–10 of 139 posts
Re: When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
#2This seems to be the end of the road for LLM's. There's only so much accuracy on a highly non-linear space you can get from a regression.
If the Pareto rule is any indicator, 80% of results come from 20% of causes. It seems that we have alot more to learn about intelligence.
I am reminded of a statement on truth from an ancient philosopher, this sentiment seems to be exactly the opposite of the LLM training paradigm
“The seeker after truth is not one who studies the writings of the ancients and, following his natural disposition, puts his trust in them, but rather the one who suspects his faith in them and questions what he gathers from them, the one who submits to argument and demonstration and not the sayings of human beings whose nature is fraught with all kinds of imperfection and deficiency. Thus the duty of the man who investigates the writings of scientists, if learning the truth is his goal, is to make himself an enemy of all that he reads, and, applying his mind to the core and margins of of its content, attack it from every side. he should also suspect himself as he performs his critical examination of it, so that he may avoid falling into either prejudice or leniency.” - ibn al-Haytham
Re: When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
#3This seems to be the end of the road for LLM's. There's only so much accuracy on a highly non-linear space you can get from a regression. If the Pareto rule is any indicator, 80% of results come from 20% of causes. It seems that we have alot more to learn about intelligence. I am reminded of a statement on truth from an ancient philosopher, this sentiment seems to be exactly the opposite of the LLM training paradigm…
I am wondering whether the reason he needed to say it was because he was arguing with those who did out their trust in the writings of the ancients.
Re: When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
#4This seems to be the end of the road for LLM's. There's only so much accuracy on a highly non-linear space you can get from a regression. If the Pareto rule is any indicator, 80% of results come from 20% of causes. It seems that we have alot more to learn about intelligence. I am reminded of a statement on truth from an ancient philosopher, this sentiment seems to be exactly the opposite of the LLM training paradigm…
Do you draw that conclusion from the fact that AI surprisingly quickly reaches the end of each ruler we try to measure it with?
Re: When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
#5Slop
> We find that nearly half of the our bench- marks exhibit saturation
Re: When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
#6Slop > We find that nearly half of the our bench- marks exhibit saturation
This leads me to believe it's NOT slop lol
Re: When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
#7Slop > We find that nearly half of the our bench- marks exhibit saturation
It's a grammatical error, sure, where is the indication of slop?
Re: When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
#8This seems to be the end of the road for LLM's. There's only so much accuracy on a highly non-linear space you can get from a regression. If the Pareto rule is any indicator, 80% of results come from 20% of causes. It seems that we have alot more to learn about intelligence. I am reminded of a statement on truth from an ancient philosopher, this sentiment seems to be exactly the opposite of the LLM training paradigm…
I really don’t think we know enough about what „intelligence“ is or how LLMs actually work to confidently say that this is the end of the road for LLM.
Re: When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
#9This seems to be the end of the road for LLM's. There's only so much accuracy on a highly non-linear space you can get from a regression. If the Pareto rule is any indicator, 80% of results come from 20% of causes. It seems that we have alot more to learn about intelligence. I am reminded of a statement on truth from an ancient philosopher, this sentiment seems to be exactly the opposite of the LLM training paradigm…
Do you draw that conclusion from the fact that AI surprisingly quickly reaches the end of each ruler we try to measure it with?
Can't call it AI like that without discrediting yourself. You mean LLMs?
Re: When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
#10This seems to be the end of the road for LLM's. There's only so much accuracy on a highly non-linear space you can get from a regression. If the Pareto rule is any indicator, 80% of results come from 20% of causes. It seems that we have alot more to learn about intelligence. I am reminded of a statement on truth from an ancient philosopher, this sentiment seems to be exactly the opposite of the LLM training paradigm…
Do you draw that conclusion from the fact that AI surprisingly quickly reaches the end of each ruler we try to measure it with?
Its not really that surprising when models are trained on the exams