Orca 2: Teaching Small Language Models How to Reason
1–10 of 82 posts
Re: Orca 2: Teaching Small Language Models How to Reason
#2Orca 2-13B consistently beat Llama 2-70B on most benchmarks in 0-shot. Hopefully, research papers will start to include Mistral/Zephyr 7B & Openchat 3.5. Even though they're smaller, they're getting competitive against much larger models and they're much cheaper to orchestrate.
Re: Orca 2: Teaching Small Language Models How to Reason
#3It fails other benchmarks vs Mistral-7b.
https://twitter.com/Teknium1/status/1726846755344634020
(There is some doubts about the validity of the comparaison in the comments)
Re: Orca 2: Teaching Small Language Models How to Reason
#4For smaller models, I'm impressed by Mistral-7b or fine-tuned variants like Zephyr. I use it regularly in Neovim[1] for mundane tasks (grammar correction, summaries, ...). I'm curious how Orca 2 performs, downloading it right now.
[1]: with https://github.com/David-Kunz/gen.nvim
Re: Orca 2: Teaching Small Language Models How to Reason
#5For smaller models, I'm impressed by Mistral-7b or fine-tuned variants like Zephyr. I use it regularly in Neovim[1] for mundane tasks (grammar correction, summaries, ...). I'm curious how Orca 2 performs, downloading it right now. [1]: with https://github.com/David-Kunz/gen.nvim
I'd love to see some demo of that!
Re: Orca 2: Teaching Small Language Models How to Reason
#6For smaller models, I'm impressed by Mistral-7b or fine-tuned variants like Zephyr. I use it regularly in Neovim[1] for mundane tasks (grammar correction, summaries, ...). I'm curious how Orca 2 performs, downloading it right now. [1]: with https://github.com/David-Kunz/gen.nvim
I'd love to see some demo of that!
A demo video is in the README (I used Mistral-7b in there).
Re: Orca 2: Teaching Small Language Models How to Reason
#7For smaller models, I'm impressed by Mistral-7b or fine-tuned variants like Zephyr. I use it regularly in Neovim[1] for mundane tasks (grammar correction, summaries, ...). I'm curious how Orca 2 performs, downloading it right now. [1]: with https://github.com/David-Kunz/gen.nvim
Also OpenChat-3.5v model (It has 7B parameters, I think it is also a Mistral finetuning), demo: https://openchat.team/
Re: Orca 2: Teaching Small Language Models How to Reason
#8I really really want this to work.
However at this point - benchmark success is about as effective as results from someone who has been “taught the test”
If say… Merck wanted to use this same model to reason out a logistics issue, or apply it to some business problem at scale - you’d have to deal with hallucinations all over the place.
The best analogy I have right now is that improved results on benchmarks are like better acting from Hugh Laurie as House.
If you want to watch a show - great (generative work)
If you want to get a prescription - then not so much.
Re: Orca 2: Teaching Small Language Models How to Reason
#9I really really want this to work. However at this point - benchmark success is about as effective as results from someone who has been “taught the test” If say… Merck wanted to use this same model to reason out a logistics issue, or apply it to some business problem at scale - you’d have to deal with hallucinations all over the place. The best analogy I have right now is that improved results on benchmarks are like…
I'm not a real AI doctor, I just play one on chat.openai.com.
Re: Orca 2: Teaching Small Language Models How to Reason
#10Are we beginning to see "specialized SLMs"? We've already seen some pretend-agent based solutions (where the same model is given several different roles and made to act as eg. ceo / architect / dev / sales in a startup).
I wonder if the way forward is to train smaller models with different sets of "skills" or "neural affinities". One for reasoning, one for summarization, one for math, one for code, etc - then combining them into full-fledged solutions. Perhaps smaller models can be "better" at their specific domains/tasks than the giant generalist models can be at any of them.