If I go to Bard, it specifically says that it' PaLM2 (on the side).
Gemini AI
391–400 of 1001 posts
Re: Gemini AI
#392This was all chosen to be able to fold in to the q4 earnings cutoff to close before end of q4-2023.
Remember it’s all a dog and pony show for shareholders.
Re: Gemini AI
#393I'm specifically asking bard if it's running on top of Gemini.
The answer is no which clearly contradicts the content of the blog post.
Another excellently planned launch by Google.
Re: Gemini AI
#394To test whether bard.google.com is already updated in your region, this prompt seems to work: Which version of Bard am I using? Here in Europe (Germany), I get: The current version is Bard 2.0.3. It is powered by the Google AI PaLM 2 model Considering that you have to log in to use Bard while Bing offers GPT-4 publicly and that Bard will be powered by Gemini Pro, which is not the version that they say beats GPT-4, it…
It's probably hallucinating that versioning. You can't trust LLMs to provide info about themselves.
Bard doesn't support Gemini in Europe yet
Re: Gemini AI
#395Bard now is pretty fast & gives pretty good code answers. I haven't been able to use Claude in EU, but I can actually use this for work, not GPT-4 level, but impressive. Looking forward to try Ultra. One thing I like from GPT, even though it's overall slower, is that you see it typing, this allows you to already process things and see if it's going in the right direction.
Bard has a setting to enable something approximating streaming responses (still not quite as nice as GPT)
Re: Gemini AI
#396Bard still not available in Canada so i can't use it ¯\_(ツ)_/¯. Wonder why Google is the only one that can't release their model here.
There are also countries like the uk that get bard but without gemini but with the old palm model instead.
https://support.google.com/bard/answer/14294096?visit_id=638...
Re: Gemini AI
#397For others that were confused by the Gemini versions: the main one being discussed is Gemini Ultra (which is claimed to beat GPT-4). The one available through Bard is Gemini Pro . For the differences, looking at the technical report [1] on selected benchmarks, rounded score in %: Dataset | Gemini Ultra | Gemini Pro | GPT-4 MMLU | 90 | 79 | 87 BIG-Bench-Hard | 84 | 75 | 83 HellaSwag | 88 | 85 | 95 Natural2Code | 75 |…
formatted nicely: Dataset | Gemini Ultra | Gemini Pro | GPT-4 MMLU | 90 | 79 | 87 BIG-Bench-Hard | 84 | 75 | 83 HellaSwag | 88 | 85 | 95 Natural2Code | 75 | 70 | 74 WMT23 | 74 | 72 | 74
I realize that this is essentially a ridiculous question, but has anyone offered a qualitative evaluation of these benchmarks? Like, I feel that GPT-4 (pre-turbo) was an extremely powerful model for almost anything I wanted help with. Whereas I feel like Bard is not great. So does this mean that my experience aligns with "HellaSwag"?
Re: Gemini AI
#398One of my biggest concerns with many of these benchmarks is that it’s really hard to tell if the test data has been part of the training data. There are terabytes of data fed into the training models - entire corpus of internet, proprietary books and papers, and likely other locked Google docs that only Google has access to. It is fairly easy to build models that achieve high scores in benchmarks if the test data has…
Everyone in the open source LLM community know the standard benchmarks are all but worthless. Cheating seems to be rampant, and by cheating I mean training on test questions + answers. Sometimes intentional, sometimes accidental. There are some good papers on checking for contamination, but no one is even bothering to use the compute to do so. As a random example, the top LLM on the open llm leaderboard right now has…
The issue is you really need to create a brand new benchmark with each release.
Users will invariably test variants of existing benchmarks/questions and thus they will be included in the next training run.
Academia isn't used to using novel benchmark questions every few months so will have trouble adapting.
Re: Gemini AI
#399Bard now is pretty fast & gives pretty good code answers. I haven't been able to use Claude in EU, but I can actually use this for work, not GPT-4 level, but impressive. Looking forward to try Ultra. One thing I like from GPT, even though it's overall slower, is that you see it typing, this allows you to already process things and see if it's going in the right direction.
If you're in Europe, Bard doesn't support Gemini yet
Re: Gemini AI
#400Earlier quoted context omitted.
The table is *highly* misleading. It uses different methodologies all over the place. For MMLU, it highlights the CoT @ 32 result, where Ultra beats GPT4, but it loses to GPT4 with 5-shot, for example. For GSM8K it uses Maj1@32 for Ultra and 5-shot CoT for GPT4, etc. Then also, for some reason, it uses different metrics for Ultra and Pro, making them hard to compare. What a mess of a "paper".
It really feels like the reason this is being released now and not months ago is that that's how long it took them to figure out the convoluted combination of different evaluation procedures to beat GPT-4 on the various benchmarks.
This, and also building the marketing website.
It feels really desperate