Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An In-depth Look at Gemini's Language Abilities

arxiv.org

Post reply on HN
Print view View original

31–40 of 73 posts

Re: An In-depth Look at Gemini's Language Abilities

#31

Post by SeanAnderson »

paxys wrote: post #3

Has anyone (outside of Google) gotten to play with Gemini Ultra yet? Been hearing a lot about Pro, but I'd be interested in seeing whether Ultra is really close to as capable as they claim. Also very interesting that Mixtral 8x7B ranks in the same neighborhood as Gemini Pro/GPT 3.5 Turbo/Claude 2.1 while being fully open source and Apache 2.0 licensed.

The hosts of the All In podcast have used it, but they're billionaires. They think highly of Ultra. Early on they just talk about the paper that's released, but then they drop that they've used Ultra.

https://www.youtube.com/watch?v=IeKUcpU5-Xk&t=3667s

Re: An In-depth Look at Gemini's Language Abilities

#32

Post by GaggiX »

thomasahle wrote: post #24
GaggiX wrote: post #18

Earlier quoted context omitted.

These are the rules of the battle arena: -Ask any question to two anonymous models (e.g., ChatGPT, Claude, Llama) and vote for the better one! -You can continue chatting until you identify a winner. -Vote won’t be counted if model identity is revealed during conversation.

It's not completely blind/anonymous, since you can just ask "What's your name" and the model will identify itself. Edit: I missed the third rule. I wonder how smart their detection is.

That's why the third rule exists.

Re: An In-depth Look at Gemini's Language Abilities

#33

Post by helsinkiandrew »

One thing this doesn't cover is speed/latency of response. Some of the things I'm doing on ChatGPT could probably work on a 90% capable GPT3.5 and other factors become more of an issue.

If I was already using GCP and they reduced their price (>10%) and offered tight integration with rest of GCP services it would still be appealing.

Re: An In-depth Look at Gemini's Language Abilities

#34

Post by _giorgio_ »

unstuck3958 wrote: post #2

It's incredible how accurate the Chatbot Arena Leaderboard [0] is at predicting model performance compared to benchmarks (which can and are being gamed, see all the 7B models on HF leaderboard) [0]: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

I've used the Arena a lot, and the differences between models are very clear 90% of the times.

I only make technical (pytorch) questions though.

Re: An In-depth Look at Gemini's Language Abilities

#35

Post by n2d4 »

euazOn wrote: post #26
n2d4 wrote: post #12

Earlier quoted context omitted.

Mixtral is on-par with Gemini Pro, not Gemini Ultra (and even there it is further behind Gemini Pro than Gemini Pro is behind GPT 3.5). But to directly answer your question, they are quite well-funded, having raised over $700mil to date. I definitely wouldn't count them out.

Gemini Ultra is not out yet. With the same logic, you could compare an unreleased Mistral model with Gemini Ultra.

Right. I'm just pointing out that comparing one model with a distilled version of another and then making broad statements about the companies behind them isn't really useful.

Surely you could make a comparison of two unreleased models, but it wouldn't be interesting because you don't have any real data (and benchmarks don't really mean anything).

Re: An In-depth Look at Gemini's Language Abilities

#36

Post by sidibe »

we_love_idf wrote: post #22

I don't understand why people keep falling for Google's ad campaign. Google have its lead in AI playing video games and board games. It is cool, entertaining and all that jazz. But OpenAI and MS are the real leaders in real AI.

Even if you don't think Google doesn't have a talent or product chops to be leaders in AI, Google can do things cheaper than others because of their infrastructure. When they do release something useful they'll probably be able to offer it free and force it on people on the most visited pages/most used browser. Surprised how many people think having a years head start means OpenAI and Microsoft are going to always be ahead

Re: An In-depth Look at Gemini's Language Abilities

#37

Post by pixl97 »

cfiggers wrote: post #27
mkmk wrote: post #9

Earlier quoted context omitted.

Mixtral is a mystery to me. How in the world is that team on par with/beating GOOGLE, who presumably have all the resources in the world to throw at this?

There's a survivorship bias going on here. You've never heard of the thousands of teams out there that are Mistral's size but AREN'T getting results that compete on the global stage, but they do exist. But you've heard of Google, whether they're getting it right or not.

I wonder how much GPU compute time is effectively being discarded by these companies as the results turn out to be garbage?

Re: An In-depth Look at Gemini's Language Abilities

#38

Post by bitshiftfaced »

3abiton wrote: post #20
unstuck3958 wrote: post #2

It's incredible how accurate the Chatbot Arena Leaderboard [0] is at predicting model performance compared to benchmarks (which can and are being gamed, see all the 7B models on HF leaderboard) [0]: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Thanks for the reference I was searching for a benchmark that can quantify the typical user experience, as most synthetic ones are completly ineffective. At what sample size the ranking become significant? Or is it baked in the metrics (ELO)?

Elo converges on stable scores fairly quickly, depending on the K-factor. I wouldn't think it would be much of an issue at all for something like this, since you can ensure you're testing against every other member (avoiding "Elo islands"). But obviously the more trials the better.

The Glicko rating system is very similar to Elo, but it also models the variance of a given rating. It can directly tell you a "rating deviation."

Re: An In-depth Look at Gemini's Language Abilities

#39

Post by jacquesm »

sidibe wrote: post #36
we_love_idf wrote: post #22

I don't understand why people keep falling for Google's ad campaign. Google have its lead in AI playing video games and board games. It is cool, entertaining and all that jazz. But OpenAI and MS are the real leaders in real AI.

Even if you don't think Google doesn't have a talent or product chops to be leaders in AI, Google can do things cheaper than others because of their infrastructure. When they do release something useful they'll probably be able to offer it free and force it on people on the most visited pages/most used browser. Surprised how many people think having a years head start means OpenAI and Microsoft are going to always be…

> they'll probably be able to offer it free

Why do you way they will 'probably' do that? Do you have any information to back that up or is this your speculation?

Re: An In-depth Look at Gemini's Language Abilities

#40

Post by zktruth »

we_love_idf wrote: post #23
jimsimmons wrote: post #4

The Gemini white paper reports higher scores on HumanEval and other tasks. So one of Google lied, this eval has bugs, they borked the deployment is true

Most likely Google has lied. AI playing video games and board games don't translate to real world applications. Many people fail to see that.

In what respect is generating text a better predictor of real world applicability than the ability to achieve goals in a complex simulated environment containing other agents?
Post reply on HN