Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google's Bard shows big leap on LLM performance leaderboard

twitter.com

Post reply on HN
Print view View original

71–80 of 96 posts

Re: Google's Bard shows big leap on LLM performance leaderboard

#71

Post by karmasimida »

random3 wrote: post #8

I'm curious about how the benchmark is done. I suspect it can be improved in order to represent user's / usability expectations. I gave Bard a go, after seeing Jeff Dean's tweet. It's just as frustrating as it was, compared to GPT-4. It's simply off the question and unable to realize it's off. I asked it to generate a chart and 3 times it came back with "here's a chart" with no chart, finally saying it doesn't have t…

I tried Bard today, it did improve surprisingly

Disclaimers: I don’t work for Google

Re: Google's Bard shows big leap on LLM performance leaderboard

#72

Post by jxy »

j-b wrote: post #55
jxy wrote: post #49

Earlier quoted context omitted.

how odd! What exactly is lmsys using? Some hidden API that google give them so they can have a better ranking there?

Most likely through this platform: https://console.cloud.google.com/vertex-ai

Thanks. I managed to google and get two different API endpoints.

From the vertex ai:

    API_ENDPOINT="us-central1-aiplatform.googleapis.com"
    PROJECT_ID="test00"
    MODEL_ID="gemini-pro"
    LOCATION_ID="us-central1"
    
    curl \
    -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    "https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION_ID}/publishers/google/models/${MODEL_ID}:streamGenerateContent" -d '@request.json'
and from the makersuite:

    curl \
      -X POST https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key=${API_KEY} \
      -H 'Content-Type: application/json' \
      -d '@request.json'

Re: Google's Bard shows big leap on LLM performance leaderboard

#73

Post by laborcontract »

tempsy wrote: post #20

Bard is far less moderated than GPT-4 eg far fewer requests feel restricted. That alone makes it far better than GPT-4.

Except when you run afoul of bard verus gpt-4, you get banned from the rest of your livelihood versus just openai?

Re: Google's Bard shows big leap on LLM performance leaderboard

#74

Post by j-b »

jxy wrote: post #72
j-b wrote: post #55

Earlier quoted context omitted.

Most likely through this platform: https://console.cloud.google.com/vertex-ai

Thanks. I managed to google and get two different API endpoints. From the vertex ai: API_ENDPOINT="us-central1-aiplatform.googleapis.com" PROJECT_ID="test00" MODEL_ID="gemini-pro" LOCATION_ID="us-central1" curl \ -X POST \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "Content-Type: application/json" \ "https://${API_ENDPOINT}/v1/projects/${PROJECT_ID}/locations/${LOCATION_ID}/publishers/google/mo…

Created a simple app to test Gemini here:

https://github.com/dssjon/gemini/blob/main/app.py

Re: Google's Bard shows big leap on LLM performance leaderboard

#75

Post by modeless »

brokensegue wrote: post #3

this leaderboard seems easily cheated/gamed. once enough eyes are on it it will be worthless

How would you game it? I think it is clearly the least gameable leaderboard we have. A more valid criticism might be that you don't like the metric it's measuring, but I think it is a useful metric, though certainly not the only useful metric.

Re: Google's Bard shows big leap on LLM performance leaderboard

#76

Post by modeless »

modeless wrote: post #9

Wow. I've suspected for a while that Bard's performance has been limited mostly by cost. Google isn't charging for Bard and they didn't want to run a gigantic model for everyone for free forever. Maybe they made a breakthrough in inference cost for their better models? Or maybe they got tired of everyone clowning on them for being behind and decided to eat the cost for a while. I still think they ought to launch a su…

New information from a Google employee: this new leaderboard entry (Bard - Gemini Pro) is a different fine-tune than the previous one (Gemini Pro - Dev API), but more importantly it "has access to the Internet" which I assume means it uses Google Search when generating answers. I bet this is responsible for the boost!

Does anyone know if the GPT-4 Turbo version used on the leaderboard has access to web search? I always assumed it did not, but now it doesn't seem like an apples-to-apples comparison.

https://x.com/asadovsky/status/1750983142041911412?s=20

Edit: I used the "Direct Chat" feature on lmsys to ask Bard and GPT-4 Turbo "What is the current price of Bitcoin?". Sure enough GPT-4 Turbo said it can't browse the Internet and Bard gave a real time answer from Google Search. This means GPT-4 outperforms Bard overall even without the ability to browse the web at all. Pretty impressive.

These seem like different categories; one is a model and one is a system with a model plus tools. I think it is useful to compare them, since there is a real difference in user experience. However, they ought to be prominently marked as different categories. And the lmsys guys ought to put a ChatGPT model on the leaderboard with its own search integration enabled, for a fairer comparison. And it would be cool to have other LLM+tools entries like Perplexity, Phind, etc.

Re: Google's Bard shows big leap on LLM performance leaderboard

#77

Post by pptr »

modeless wrote: post #75
brokensegue wrote: post #3

this leaderboard seems easily cheated/gamed. once enough eyes are on it it will be worthless

How would you game it? I think it is clearly the least gameable leaderboard we have. A more valid criticism might be that you don't like the metric it's measuring, but I think it is a useful metric, though certainly not the only useful metric.

I think there is a theoretical threat for models hosted on an external server.

You could ask a question on lmsys, check your server logs for the generated response, go back to lmsys and pick the response that your model generated.

Maybe you could also use a better model for requests from lmsys. E. g. use an unquantized model, disable censorship, etc.

I doubt any of the big players are doing that, but you never know.

Re: Google's Bard shows big leap on LLM performance leaderboard

#78

Post by joshuanapoli »

refulgentis wrote: post #12
lumuba wrote: post #5

how is GPT4-Turbo higher than GPT-4?

It's actually better. All the people claiming unannounced updates make OpenAI model significantly worse were misled by their own lying eyes. It's so hard to believe, I know, but it's true.

It depends on the size of the response that you need…

Re: Google's Bard shows big leap on LLM performance leaderboard

#79

Post by qeternity »

refulgentis wrote: post #63
qeternity wrote: post #58

Earlier quoted context omitted.

It isn’t, because performance isn’t scalar. It seems to be a more preferable chatbot in this arena. It is objectively less capable for many other domain specific tasks.

How does the arena work? ;)

Lol it’s literally called the “LMSYS Chatbot Arena”.

But yeah we tried some things.

Re: Google's Bard shows big leap on LLM performance leaderboard

#80

Post by qeternity »

kccqzy wrote: post #59
qeternity wrote: post #57

Earlier quoted context omitted.

Google is fighting for its life here. They are not worried about cost.

Google is one of the most cost conscious companies in tech when it comes to compute costs. Sure they are blind to other kinds of cost like reputation damage due to stupid leadership decisions. But in terms of their tech, they run their servers and their network to very high utilization, definitely exceeding competitors like Amazon.

You mention cost, and then proceed to describe efficiency. They are different things.

They can spend billions giving away compute (cost) and be extremely efficient (high utilization) in doing so.

Post reply on HN