Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What can LLMs never do?

strangeloopcanon.com

Post reply on HN
Print view View original

21–30 of 385 posts

Re: What can LLMs never do?

#21

Post by Xenoamorphous »

vsnf wrote: post #20
Xenoamorphous wrote: post #18

There’s many things they can’t do. Even a simple rule like “ensure that numbers from one to ten are written as words and numbers greater ten as digits in the given text” fails for me for so many examples even if it works for many others; few shot, chain of thought, many versions of the prompt, it doesn’t matter. Sometimes LLMs will even change the number to something else, even with temp set to 0. And then there’s th…

> ensure that numbers from one to ten as written as words and numbers greater ten as digits in the given text I can’t fault llms for not knowing what to do here because I, a human, have no idea what on earth this means.

“Ten” is a word, “10” are digits.

I’m not a native English speaker, how would you write it?

FWIW the LLMs get it right many times, but fail other times.

Re: What can LLMs never do?

#22

Post by vouaobrasil »

rmellow wrote: post #17
vouaobrasil wrote: post #10

[flagged]

Technology amplifies intent. LLMs are as friendly to society as nuts and bolts. It's up to those who wield it to be moral.

Not true. Technology defines the parameters of social action and we are forced to use technology as it becomes mandatory. Moreover, humans have basic instincts, the strong force which overrides morality frequently. Humanity as a society has very little will and a lot of momentum that is amplified by technology. It is not up to anyone to wield anything.

Re: What can LLMs never do?

#23

Post by [deleted] »

vsnf wrote: post #20
Xenoamorphous wrote: post #18

There’s many things they can’t do. Even a simple rule like “ensure that numbers from one to ten are written as words and numbers greater ten as digits in the given text” fails for me for so many examples even if it works for many others; few shot, chain of thought, many versions of the prompt, it doesn’t matter. Sometimes LLMs will even change the number to something else, even with temp set to 0. And then there’s th…

> ensure that numbers from one to ten as written as words and numbers greater ten as digits in the given text I can’t fault llms for not knowing what to do here because I, a human, have no idea what on earth this means.

[deleted]

Re: What can LLMs never do?

#24

Post by stavros »

gwervc wrote: post #19

> But then I started asking myself how can we figure out the limits of its ability to reason Third paragraph. The entire article is based on the premise LLMs are supposed to reason, which is wrong. They don't, they're tools to generate text.

I really hate this reductive, facile, "um akshually" take. If the text that the text-generating tool generates contains reasoning, then the text generation tool can be said to be reasoning, can't it.

That's like saying "humans aren't supposed to reason, they're supposed to make sounds with their mouths".

Re: What can LLMs never do?

#25

Post by Snacklive »

vsnf wrote: post #20
Xenoamorphous wrote: post #18

There’s many things they can’t do. Even a simple rule like “ensure that numbers from one to ten are written as words and numbers greater ten as digits in the given text” fails for me for so many examples even if it works for many others; few shot, chain of thought, many versions of the prompt, it doesn’t matter. Sometimes LLMs will even change the number to something else, even with temp set to 0. And then there’s th…

> ensure that numbers from one to ten as written as words and numbers greater ten as digits in the given text I can’t fault llms for not knowing what to do here because I, a human, have no idea what on earth this means.

Given the text "1,2,3,4,5,6,7,8,9,10,11,12" it should result in "one, two, three, four, five, six, seven, eight, nine, ten, 11, 12"

or at least that's my understanding of the prompt

Re: What can LLMs never do?

#26

Post by callmeal »

vsnf wrote: post #20
Xenoamorphous wrote: post #18

There’s many things they can’t do. Even a simple rule like “ensure that numbers from one to ten are written as words and numbers greater ten as digits in the given text” fails for me for so many examples even if it works for many others; few shot, chain of thought, many versions of the prompt, it doesn’t matter. Sometimes LLMs will even change the number to something else, even with temp set to 0. And then there’s th…

> ensure that numbers from one to ten as written as words and numbers greater ten as digits in the given text I can’t fault llms for not knowing what to do here because I, a human, have no idea what on earth this means.

> > ensure that numbers from one to ten as written as words and numbers greater ten as digits in the given text

There are two blue, one red, and 15 green m&ms in this bag.

Re: What can LLMs never do?

#28

Post by FroshKiller »

vsnf wrote: post #20
Xenoamorphous wrote: post #18

There’s many things they can’t do. Even a simple rule like “ensure that numbers from one to ten are written as words and numbers greater ten as digits in the given text” fails for me for so many examples even if it works for many others; few shot, chain of thought, many versions of the prompt, it doesn’t matter. Sometimes LLMs will even change the number to something else, even with temp set to 0. And then there’s th…

> ensure that numbers from one to ten as written as words and numbers greater ten as digits in the given text I can’t fault llms for not knowing what to do here because I, a human, have no idea what on earth this means.

It's a simple prescriptive rule in English. If you are writing about a small number, like less than ten, spell it out. For example: "According to a survey, nine out of ten people agree."

But if you are writing about a large number, particularly one with a lot of different digits, prefer writing the digits: "A mile is 5,280 feet." Compare that to: "A mile is five thousand, two hundred, and eighty feet."

Re: What can LLMs never do?

#30

Post by logtempo »

vsnf wrote: post #20
Xenoamorphous wrote: post #18

There’s many things they can’t do. Even a simple rule like “ensure that numbers from one to ten are written as words and numbers greater ten as digits in the given text” fails for me for so many examples even if it works for many others; few shot, chain of thought, many versions of the prompt, it doesn’t matter. Sometimes LLMs will even change the number to something else, even with temp set to 0. And then there’s th…

> ensure that numbers from one to ten as written as words and numbers greater ten as digits in the given text I can’t fault llms for not knowing what to do here because I, a human, have no idea what on earth this means.

I think he mean that numbers less or equal than ten are written as words, and others are written as numbers.

Given the many reaponses, it would be fun to aee if llm beat humans on understanding the sentence ahah

Post reply on HN