Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Orca 2: Teaching Small Language Models How to Reason

arxiv.org

Post reply on HN
Print view View original

21–30 of 82 posts

Re: Orca 2: Teaching Small Language Models How to Reason

#21

Post by FFP999 »

intended wrote: post #8

I really really want this to work. However at this point - benchmark success is about as effective as results from someone who has been “taught the test” If say… Merck wanted to use this same model to reason out a logistics issue, or apply it to some business problem at scale - you’d have to deal with hallucinations all over the place. The best analogy I have right now is that improved results on benchmarks are like…

At the moment I read "how to reason" in the headline my bullshit detector started to go off.

LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.

Re: Orca 2: Teaching Small Language Models How to Reason

#22

Post by Chabsff »

yujian wrote: post #20

I'm not sure if I'm missing something from the paper, but are multi-billion parameter models getting called "small" language models now? And when did this paradigm shift happen?

Nowadays, small essentially means realistically useable on prosumer hardware.

Re: Orca 2: Teaching Small Language Models How to Reason

#24

Post by hmottestad »

yujian wrote: post #20

I'm not sure if I'm missing something from the paper, but are multi-billion parameter models getting called "small" language models now? And when did this paradigm shift happen?

All the llama models, including the 70B one can run on consumer hardware. You might be able to fit GPT-3 (175B) at Q4 or Q3 on a Mac Studio, but that's probably the limit for consumer hardware. At 4-bit a 7B model requires some 4GB of ram, so that should probably be possible to run on a phone, just not very fast.

Re: Orca 2: Teaching Small Language Models How to Reason

#25

Post by coderaptor »

FFP999 wrote: post #21
intended wrote: post #8

I really really want this to work. However at this point - benchmark success is about as effective as results from someone who has been “taught the test” If say… Merck wanted to use this same model to reason out a logistics issue, or apply it to some business problem at scale - you’d have to deal with hallucinations all over the place. The best analogy I have right now is that improved results on benchmarks are like…

At the moment I read "how to reason" in the headline my bullshit detector started to go off. LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.

I haven’t heard a definition of “reasoning” or “thinking” that proves humans aren’t doing exactly that same probabilistic regurgitation.

I don’t think it’s possible to prove; feels like a philosophical question.

Re: Orca 2: Teaching Small Language Models How to Reason

#26

Post by moffkalast »

yujian wrote: post #20

I'm not sure if I'm missing something from the paper, but are multi-billion parameter models getting called "small" language models now? And when did this paradigm shift happen?

When 175B, 300B, 1.8T models are considered large, 7B is considered small.

Re: Orca 2: Teaching Small Language Models How to Reason

#27

Post by QuadmasterXLII »

FFP999 wrote: post #21
intended wrote: post #8

I really really want this to work. However at this point - benchmark success is about as effective as results from someone who has been “taught the test” If say… Merck wanted to use this same model to reason out a logistics issue, or apply it to some business problem at scale - you’d have to deal with hallucinations all over the place. The best analogy I have right now is that improved results on benchmarks are like…

At the moment I read "how to reason" in the headline my bullshit detector started to go off. LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.

With only the information we had in 2020, the two theories “language models don’t reason, they regurgitate” and “as language models scale, they begin to think and reason” made predictions, and the people who invested time and money based on the predictions of the latter theory have done well for themselves.

Re: Orca 2: Teaching Small Language Models How to Reason

#28

Post by FFP999 »

QuadmasterXLII wrote: post #27
FFP999 wrote: post #21

Earlier quoted context omitted.

At the moment I read "how to reason" in the headline my bullshit detector started to go off. LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.

With only the information we had in 2020, the two theories “language models don’t reason, they regurgitate” and “as language models scale, they begin to think and reason” made predictions, and the people who invested time and money based on the predictions of the latter theory have done well for themselves.

If you're trying to tell me there's a sucker born every minute, I knew that.

Re: Orca 2: Teaching Small Language Models How to Reason

#29

Post by intended »

coderaptor wrote: post #25
FFP999 wrote: post #21

Earlier quoted context omitted.

At the moment I read "how to reason" in the headline my bullshit detector started to go off. LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.

I haven’t heard a definition of “reasoning” or “thinking” that proves humans aren’t doing exactly that same probabilistic regurgitation. I don’t think it’s possible to prove; feels like a philosophical question.

It’s possible to prove.

Use an LLM to do a real world task that you should be able to achieve by reasoning.

Re: Orca 2: Teaching Small Language Models How to Reason

#30

Post by intended »

QuadmasterXLII wrote: post #27
FFP999 wrote: post #21

Earlier quoted context omitted.

At the moment I read "how to reason" in the headline my bullshit detector started to go off. LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.

With only the information we had in 2020, the two theories “language models don’t reason, they regurgitate” and “as language models scale, they begin to think and reason” made predictions, and the people who invested time and money based on the predictions of the latter theory have done well for themselves.

The people who bet on generative tasks, are getting mileage out of tit.

People who bet on reasoning tasks, not so much.

Post reply on HN