Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Study identifies weaknesses in how AI systems are evaluated

oii.ox.ac.uk

Post reply on HN
Print view View original

151–160 of 204 posts

Re: Study identifies weaknesses in how AI systems are evaluated

#151

Post by [deleted] »

SV_BubbleTime wrote: post #134
sroussey wrote: post #132

Earlier quoted context omitted.

Actually… ask gpt1 to multiply 1765x9392.

I wish this was more broadly, explained to people… There are LLMs, the engines that make these products run, and then the products themselves. GPT anything should not be asked math problems. LLMs are language models, not math. The line is going to get very blurry because ChatGPT, or Claude or Gemini, are not LLM’s. Their products driven by LLMs. The question or requisite should not be can my LLM do math. It can I bui…

[deleted]

Re: Study identifies weaknesses in how AI systems are evaluated

#152

Post by stogot »

bjackman wrote: post #57
bofadeez wrote: post #38

Earlier quoted context omitted.

Even a p-value is insufficient. Maybe can use some of this stuff https://web.stanford.edu/~swager/causal_inf_book.pdf

I have actually been thinking of hiring some training contractors to come in and teach people the basics of applied statistical inference. I think with a bit of internal selling, engineers would generally be interested enough to show up and pay attention. And I don't think we need very deep expertise, just a moderate bump in the ambient level of statistical awareness would probably go a long way. It's not like there'…

Don’t most computer science programs require this? Mine had a statistics requirement

Re: Study identifies weaknesses in how AI systems are evaluated

#153

Post by andy99 »

bubblelicious wrote: post #4

I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mes…

I also work in LLM evaluation. My cynical take is that nobody is really using LLMs for stuff, and so benchmarks are mostly just make up tasks (coding is probably the exception). If we had real specific use cases it should be easier to benchmark and know if one is better, but it’s mostly all hypothetical.

The more generous take is that you can’t benchmarks advanced intelligence very well, whether LLM or person. We don’t have good procedures for assessing a person's fit-for-purpose e.g. for a job, certainly not standardized question sets. Why would we expect to be able to do this with AI?

I think both of these takes are present to some extent in reality.

Re: Study identifies weaknesses in how AI systems are evaluated

#154

Post by lossyalgo »

SV_BubbleTime wrote: post #134
sroussey wrote: post #132

Earlier quoted context omitted.

Actually… ask gpt1 to multiply 1765x9392.

I wish this was more broadly, explained to people… There are LLMs, the engines that make these products run, and then the products themselves. GPT anything should not be asked math problems. LLMs are language models, not math. The line is going to get very blurry because ChatGPT, or Claude or Gemini, are not LLM’s. Their products driven by LLMs. The question or requisite should not be can my LLM do math. It can I bui…

> A coworker of mine told me that GPT’s LLM can use Excel files. No, it can’t. But the tools they plugged into it can.

And there's a 50/50 chance they'll use the right tool for the job. I tried the math question above multiple times on gpt5 and it gets it right about 50% of the time. If i ask to "try again" it usually gets it on the 2nd or 3rd try. Most times that it's wrong, it's not far off but it looks deceptively accurate at first glance.

Re: Study identifies weaknesses in how AI systems are evaluated

#155

Post by cyanydeez »

bjackman wrote: post #18
bubblelicious wrote: post #4

I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mes…

For what it's worth, I work on platforms infra at a hyperscaler and benchmarks are a complete fucking joke in my field too lol. Ultimately we are measuring extremely measurable things that have an objective ground truth. And yet: - we completely fail at statistics (the MAJORITY of analysis is literally just "here's the delta in the mean of these two samples". If I ever do see people gesturing at actual proper analysi…

Id say your experience is being more monetized for growth for growth sake.

Re: Study identifies weaknesses in how AI systems are evaluated

#156

Post by scuff3d »

bubblelicious wrote: post #4

I work on LLM benchmarks and human evals for a living in a research lab (as opposed to product). I can say: it’s pretty much the Wild West and a total disaster. No one really has a good solution, and researchers are also in a huge rush and don’t want to end up making their whole job benchmarking. Even if you could, and even if you have the right background you can do benchmarks full time and they still would be a mes…

The big problem is that tech companies and journalist aren't transparent about this. They tout benchmark numbers constantly, like they're an object measure of capabilities.

Re: Study identifies weaknesses in how AI systems are evaluated

#157

Post by falcor84 »

SV_BubbleTime wrote: post #134
sroussey wrote: post #132

Earlier quoted context omitted.

Actually… ask gpt1 to multiply 1765x9392.

I wish this was more broadly, explained to people… There are LLMs, the engines that make these products run, and then the products themselves. GPT anything should not be asked math problems. LLMs are language models, not math. The line is going to get very blurry because ChatGPT, or Claude or Gemini, are not LLM’s. Their products driven by LLMs. The question or requisite should not be can my LLM do math. It can I bui…

> A coworker of mine told me that GPT’s LLM can use Excel files. No, it can’t. But the tools they plugged into it can.

It's a bit like saying that a human can't use Excel files, but when given a keyboard, mouse and monitor connected to a computer running Excel, it can. But then obviously the "Excel usage" competency is in the human; not in the tools, and a cat for example cannot use Excel proficiently however many training hours it gets and however good the keyboard is.

Taking it back to the LLMs, it is clear to me that some modern LLMs like the one running ChatGPT can be integrated with tools in a way that makes them somewhat proficient with Excel, while other simpler LLMs cannot, regardless of the tools.

Re: Study identifies weaknesses in how AI systems are evaluated

#158

Post by Terr_ »

bogwog wrote: post #93
riskable wrote: post #85

We should make a collective git repo full of every kind of annoying bug we (expert developers) can think of. Then use that to benchmark LLMs. Someone want to start? I've got a Yjs/CRDT collaborative editing bug that took like a week and a half of attempts with Claude Code (Sonnet 4.5), GPT5-codex (medium), and GLM-4.6 many, many attempts to figure out. Even then they didn't really get it... Just came up with a succes…

I can't tell how much of this is sarcasm > we (expert developers) ... > took like a week and a half of attempts with Claude Code ... What kind of expert developer wastes that much time prompting a bunch of different LLMs to end up with a workaround, instead of actually debugging and fixing the bug themselves?

To be charitable to the parent poster, I've had multi-week bugs that turned out to be a tiny change, where every test iteration took hours of compile time...

Re: Study identifies weaknesses in how AI systems are evaluated

#159

Post by Al-Khwarizmi »

stogot wrote: post #152
bjackman wrote: post #57

Earlier quoted context omitted.

I have actually been thinking of hiring some training contractors to come in and teach people the basics of applied statistical inference. I think with a bit of internal selling, engineers would generally be interested enough to show up and pay attention. And I don't think we need very deep expertise, just a moderate bump in the ambient level of statistical awareness would probably go a long way. It's not like there'…

Don’t most computer science programs require this? Mine had a statistics requirement

I don't know how it is in the US and other countries, but in my country I would say statistics is typically not taught well, at least in CS degrees. I was a very good student, always had good understanding at the subjects at university, but in the case of statistics they just taught us formulae and techniques as dogmas without much explanation of where they came from, why, and when to use them. It didn't help either that the exercises we did always applied them to things outside CS (clinical testing, people's heights and things like that) with no application we could directly relate to. As a result, when I finished the degree I had forgotten most of it, and when I started working I was surprised that it was actually useful.

When I talk about this with other CS people in my own country (Spain) they tend to refer similar experiences.

Re: Study identifies weaknesses in how AI systems are evaluated

#160

Post by Forgeties79 »

riskable wrote: post #90
Forgeties79 wrote: post #29

Earlier quoted context omitted.

>the point of these LLMs is to do things that computers were bad at. The way they’re being deployed it feels like the point of LLMs is largely to replace basic online search or to run your online customer support cheaply. I’m a bit out on a limb here because this is not really my technical expertise by any stretch of the imagination, but it seems to me these benchmark tests don’t really tell us much about how LLM’s p…

Nobody really knows "the point" of LLMs yet. They weren't even "invented" as much as they emerged as a trick to get computers to better understand human language. They're still brand spanking new and everyone's trying to figure out how to best use them. We don't even really know if they're ever going to be "really good at" any given task! Are they "really good at" these things or are they merely "OK-ish"? * Answering…

> Nobody really knows "the point" of LLMs yet

Yet literally hundreds of billions of dollars are being invested in them. That’s what’s so concerning. And I can tell you not one of these startups would EVER acknowledge the truth of your statement.

Post reply on HN