Earlier quoted context omitted.
I don't think you and him are in disagreement. I read it as him saying "evaluating LLMs is extremely difficult and a big problem right now is that many people are treating them as basically human in capability". Its the opposite problem to the perception of computers in the 70s, early computers were seen by some as too alien to be as useful as a person across most tasks, llms are seen by some as too human to not be a…
WHO exactly is treating them as human? It is a strawman.
Quite a lot of people are.