Earlier quoted context omitted.
Gpt 3.5 turbo is 20B
I doubt that. What's your source?
There was a paper published by Microsoft that seemed to leak this detail. I'm on mobile right now and don't have a link but it should be searchable
Orca 2: Teaching Small Language Models How to Reason
71–80 of 82 posts
Re: Orca 2: Teaching Small Language Models How to Reason
#72Earlier quoted context omitted.
I doubt that. What's your source?
There was a paper published by Microsoft that seemed to leak this detail. I'm on mobile right now and don't have a link but it should be searchable
The paper was https://arxiv.org/abs/2310.17680
It has been withdrawn with this note:
> Contains inappropriately sourced conjecture of OpenAI's ChatGPT parameter count from this http URL, a citation which was omitted. The authors do not have direct knowledge or verification of this information, and relied solely on this article, which may lead to public confusion
(the noted URL is a just a Forbes blogger with no special qualifications that would make what he claimed particularly credible).
Re: Orca 2: Teaching Small Language Models How to Reason
#73Earlier quoted context omitted.
At the moment I read "how to reason" in the headline my bullshit detector started to go off. LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.
Just yesterday I saw an example of a person asking GPT what "fluftable" means. The word was invented by their little daughter and they didn't know what it meant. GPT reasoned it was a portmaneau of"fluffy" and "comfortable", and it made sense because it was used in reference to a pillow. If it's just regurgitation, I'd like to know how it's able to understand novel words not found in the training data...
Because you’re not understanding what it’s regurgitating. It’s not a fact machine that regurgitates knowledge, in fact it’s not really so good at that. It regurgitates plausible patterns of language, and combining words and such is hardly a rare pattern
Re: Orca 2: Teaching Small Language Models How to Reason
#74Earlier quoted context omitted.
Just yesterday I saw an example of a person asking GPT what "fluftable" means. The word was invented by their little daughter and they didn't know what it meant. GPT reasoned it was a portmaneau of"fluffy" and "comfortable", and it made sense because it was used in reference to a pillow. If it's just regurgitation, I'd like to know how it's able to understand novel words not found in the training data...
I would read Francois Chollet's explanation of this. It's very good: https://fchollet.substack.com/p/how-i-think-about-llm-prompt... For words that are not in the model's vocabulary, like 'fluftable', the model uses a subword tokenization strategy. It breaks down the word into smaller known subunits (subwords or characters) and represents each subunit with its own vector. By understanding the context in which 'flufta…
What you've written does nothing to disabuse any reasonable person of the notion that LLMs cannot reason; if anything you've explained how LLM's reason, not that they cannot do it.
Re: Orca 2: Teaching Small Language Models How to Reason
#75Earlier quoted context omitted.
At the moment I read "how to reason" in the headline my bullshit detector started to go off. LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.
Just yesterday I saw an example of a person asking GPT what "fluftable" means. The word was invented by their little daughter and they didn't know what it meant. GPT reasoned it was a portmaneau of"fluffy" and "comfortable", and it made sense because it was used in reference to a pillow. If it's just regurgitation, I'd like to know how it's able to understand novel words not found in the training data...
Which is also within the realms of house MD vs doctor, potentially even more so.
LLMs are trained on realms of text, good performance here is not unexpected.
To put it another way - Would you hire chat GPT?
For work, you need to have more than text skills.
Re: Orca 2: Teaching Small Language Models How to Reason
#76Earlier quoted context omitted.
LLMs can be trained on all the math books in the world, starting from the easiest to the most advanced, they can regurgitate them almost perfectly, yet they won't apply the concepts in those books to their actions. I'd count the ability to learn new concepts and methods, then being able to use them as "reasoning".
Aren't there quite a few examples of LLMs giving out-of-distribution answers to stated problems? I think there are two issues with LLMs and reasoning: 1. They are single-pass and static - you "fake" short-term memory by re-feeding the questions with it answer 2. They have no real goal to achieve - one that it would split into sub-goals, plan to achieve them, estimate the returns of each, etc. As for 2. I think this i…
its those kinds of examples that make it hard to cleave a measurement of success.
Based on those kinds of results an LLM should, in theory, be able to plan, analyze and suggest improvements, without the need for human intervention.
You will see rudimentary success for this as well - however, when you push the tool further, it will stop being... "logical".
I'd refine the point to saying that you will get some low hanging fruit in terms of syntactic prediction and semantic analysis.
But when you lean ON semantic ability, the model is no longer leaning on its syntactic data set, and it fails to generalize.
Re: Orca 2: Teaching Small Language Models How to Reason
#77Official Orca-2 demo is available on huggingface Spaces now - https://huggingface.co/spaces/ari9dam/Orca-2-13B
Re: Orca 2: Teaching Small Language Models How to Reason
#78Earlier quoted context omitted.
At the moment I read "how to reason" in the headline my bullshit detector started to go off. LLMs do not reason, they do not think, they are not AGI. They generate by regurgitating.
You are simply incorrect. They can reason.
and how can you tell they reason and not parrot some text in training data?
There are papers about trying LLMs on generated reasoning problems, and they usually fail.
Re: Orca 2: Teaching Small Language Models How to Reason
#79Earlier quoted context omitted.
I asked GPT 4 and it had some counter points: Reasoning blends learned skills and natural cognition. It integrates new information, not just past memories. Reasoning is adaptable, not rigidly algorithmic. Emotions and context also shape reasoning. which seemed to make sense.
I hope this will be found in history books and some students will point the irony that people are relying on gpt4's arguments about reasoning in a thread where it's proclaimed that said model can't reason
In fact it is not absurd or weird. The model does not need to be capable of x/reasoning to produce knowledge about x/reasoning. A book with a chapter on x/reasoning doesn't reason either.
Re: Orca 2: Teaching Small Language Models How to Reason
#80Earlier quoted context omitted.
You are simply incorrect. They can reason.
and how can you tell they reason and not parrot some text in training data? There are papers about trying LLMs on generated reasoning problems, and they usually fail.
>Usually
That implies - sometimes not. Which would prove at least some reasoning capabilities.