I don't think "accuracy" is going to be the defining feature of which chatbot succeeds. People just aren't using them for tasks where a 3-5 point difference makes the grade, because the difference between 67 and 100 is more important than the difference between 64 and 67. If you can integrate a relatively speedy bot somewhere people can use it conveniently that'll get more usage than a slightly more factual response you have to tab out to.
An In-depth Look at Gemini's Language Abilities
51–60 of 73 posts
Re: An In-depth Look at Gemini's Language Abilities
#52Earlier quoted context omitted.
Mixtral is a mystery to me. How in the world is that team on par with/beating GOOGLE, who presumably have all the resources in the world to throw at this?
Mistral.AI was founded by three people from Deepmind, they're beating Google because Google no longer has them.
Slight correction, Mistral.AI was funded by two people from Meta (Guillaume Lample, Timothée Lacroix) and one from Deepmind (Arthur Mensch).
For new technologies, what matters most might be the universities where people are from, rather than the companies. The founders of Google graduated from Stanford. The founders of Mistra AI graduated from École Polytechnique and École Normale Supérieure, that are renowned in France, notably for their scientific formations.
Re: An In-depth Look at Gemini's Language Abilities
#53Earlier quoted context omitted.
Most likely Google has lied. AI playing video games and board games don't translate to real world applications. Many people fail to see that.
In what respect is generating text a better predictor of real world applicability than the ability to achieve goals in a complex simulated environment containing other agents?
The context in simulated environments of games is far less complex than the real world. Also the available interactions far less.
It would be different if the agent would be exposed to the real world and use multisensory data to predict the next "token", i.e. thought or action.
Re: An In-depth Look at Gemini's Language Abilities
#54Earlier quoted context omitted.
Most likely Google has lied. AI playing video games and board games don't translate to real world applications. Many people fail to see that.
In what respect is generating text a better predictor of real world applicability than the ability to achieve goals in a complex simulated environment containing other agents?
People pay money for it.
Re: An In-depth Look at Gemini's Language Abilities
#55I don't understand why people keep falling for Google's ad campaign. Google have its lead in AI playing video games and board games. It is cool, entertaining and all that jazz. But OpenAI and MS are the real leaders in real AI.
This is a very ignorant comment. AlphaFold is far more useful than ChatGPT and remains a more impressive piece of technology, even if it doesn't help you write boilerplate code.
Re: An In-depth Look at Gemini's Language Abilities
#56Earlier quoted context omitted.
It's astounding that Mixtral Instruct ties with 3.5-turbo while being ~10x smaller.
3.5-turbo might be 20B, not 10x larger https://www.reddit.com/r/LocalLLaMA/comments/17jrj82/new_mic...
Let's see... the linked arXiv article has been withdrawn by the author with the following comment:
> Contains inappropriately sourced conjecture of OpenAI's ChatGPT parameter count from this http URL, a citation which was omitted. The authors do not have direct knowledge or verification of this information, and relied solely on this article, which may lead to public confusion
The URL in question: https://www.forbes.com/sites/forbestechcouncil/2023/02/17/is...
This article was written by Aleks Farseev, the CEO of SoMonitor.ai, who makes the claim with no source or explanation:
> ChatGPT is not just smaller (20 billion vs. 175 billion parameters) and therefore faster than GPT-3
Re: An In-depth Look at Gemini's Language Abilities
#57I don't understand why people keep falling for Google's ad campaign. Google have its lead in AI playing video games and board games. It is cool, entertaining and all that jazz. But OpenAI and MS are the real leaders in real AI.
Even if you don't think Google doesn't have a talent or product chops to be leaders in AI, Google can do things cheaper than others because of their infrastructure. When they do release something useful they'll probably be able to offer it free and force it on people on the most visited pages/most used browser. Surprised how many people think having a years head start means OpenAI and Microsoft are going to always be…
I have been waiting for Google to release a decent translation model, but it seems they will only offer a cheap one. DeepL has been besting Google Translate for years. Why? I think Google has aversion to making large models public.
Re: An In-depth Look at Gemini's Language Abilities
#58Earlier quoted context omitted.
Most likely Google has lied. AI playing video games and board games don't translate to real world applications. Many people fail to see that.
In what respect is generating text a better predictor of real world applicability than the ability to achieve goals in a complex simulated environment containing other agents?
It's not one or the other. We need both supervised pre-training and reinforcement learning. The first part represents past human experiences encoded as language. They can bring a model to human level on most tasks, but not make it smarter.
The second approach, with RL, is based on immediate feedback and could make a model smarter than us. Just think of AlphaZero or AlphaTensor. But this requires deploying a wide search over possible solutions and using a mechanism to rank or filter the bad ideas out (code execution, running a simulation or a game, optimizing some metric)
So models need both past experience and new experience to advance. They can use organic text initially, but later need to develop their own training examples. The feedback they get will be on topic, both with the human user and with the model mistakes. That's very valuable. Feedback learning is what could make LLMs finally graduate from mediocre results.
DeepMind is saying they are using both, and feedback learning is dialed up.
Re: An In-depth Look at Gemini's Language Abilities
#59Starling 7b alpha is close to gpt3.5 but it asking list 5 capitals sorted by population size and it will run on like forever generating 100 capitals and explainations and gibrish. Not sure if it’s my prompt, but it sure isn’t close to the intelligence of gpt3.5 when asked the same. Starling: ``` USER list 5 largest capitals sorted by largest population first AI Here are some countries and their capitals in alphabetic…
Sampling methods also affect this. Have you tried min_p sampling? https://github.com/ggerganov/llama.cpp/pull/3841#issuecommen...
Re: An In-depth Look at Gemini's Language Abilities
#60Earlier quoted context omitted.
It's because it isn't "predicting" anything, but rather aggregating user feedback. That is of course going to be closest to judging the subjective "best" model that pleases most people. It's like saying how can evaluating 5 years of performance at work be better at predicting someone's competency than their SAT scores.
But, what if you could make an SAT that is equivalent to evaluating years of performance at work? https://huggingface.co/papers/2306.05685 This paper makes the argument that... "Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explaina…
My understanding was that GPT4 evaluation appeared to specifically favour text that GPT4 would generate itself (leading to some bias towards gpt-based fine-tunes), although I can't remember the details