Gemini AI • Hacker News Board

Re: Gemini AI

#321

Post by MadSudaca » Wed, Dec 06, 2023, 4:57 PM UTC

DeathArrow wrote: post #123

MadSudaca wrote: post #28

It's truly astounding to me that Google, a juggernaut with decades under its belt on all things AI, is only now catching up to OpenAI which is on all camps a fraction of its size.

Maybe small teams can be faster than huge teams?

Sure, but it doesn’t mean that it stops being surprising. It’s like a “time is relative” kind of thing for organizational logic.

Imagine an organization on the scale of Google, with everything in it’s favor, being outmaneuvered by a much smaller one in such a transcendental endeavor. It’s like to a small country in Central America, coming up with some weapon to rival the US’s army.

Re: Gemini AI

#322

Post by markdog12 » Wed, Dec 06, 2023, 4:57 PM UTC

Still can't use Bard in Canada.

Re: Gemini AI

#323

Post by TheCaptain4815 » Wed, Dec 06, 2023, 4:58 PM UTC

walthamstow wrote: post #32

Gemini Nano sounds like the most exciting part IMO. IIRC Several people in the recent Pixel 8 thread were saying that offloading to web APIs for functions like Magic Eraser was only temporary and could be replaced by on-device models at some point. Looks like this is the beginning of that.

Unless they make it open source, what benefits would it have compared to the tiny OpenAi models?

Re: Gemini AI

#324

Post by Palmik » Wed, Dec 06, 2023, 4:58 PM UTC

albertzeyer wrote: post #42

So, better than GPT4 according to the benchmarks? Looks very interesting. Technical paper: https://goo.gle/GeminiPaper Some details: - 32k context length - efficient attention mechanisms (for e.g. multi-query attention (Shazeer, 2019)) - audio input via Universal Speech Model (USM) (Zhang et al., 2023) features - no audio output? (Figure 2) - visual encoding of Gemini models is inspired by our own foundational work o…

The table is *highly* misleading. It uses different methodologies all over the place.

For MMLU, it highlights the CoT @ 32 result, where Ultra beats GPT4, but it loses to GPT4 with 5-shot, for example.

For GSM8K it uses Maj1@32 for Ultra and 5-shot CoT for GPT4, etc.

Then also, for some reason, it uses different metrics for Ultra and Pro, making them hard to compare.

What a mess of a "paper".

Re: Gemini AI

#325

Post by tbalsam » Wed, Dec 06, 2023, 4:59 PM UTC

Apparently designed for mobile inference too, I've heard the weights on the nano model were quantized down to uint4.

Will be exciting to see how all of that plays out in terms of 'LLMs on phones', going forward.

People who know me know that I can be pretty curmudgeony about a lot of various technological things, but I really think that this could be a hard core paradigm shift in terms of mobile capabilities, lol.

Like, the real story here is the next step in the evolution of the role of mobile devices in people's lives, this is one of the biggest/clearest/most official 'shotd across the bow' that one could make for something like this, I think, lol.

Re: Gemini AI

#326

Post by cchance » Wed, Dec 06, 2023, 4:59 PM UTC

obastani wrote: post #155

Important caveat with some of the results: they are using better prompting techniques for Gemini vs GPT-4, including their top line result on MMLU (CoT@32 vs top-5). But, they do have better results on zero-shot prompting below, e.g., on HumanEval.

I do find it a bit dirty to use better prompt techniques and compare them in a chart like that

Re: Gemini AI

#327

Post by Liutprand » Wed, Dec 06, 2023, 4:59 PM UTC

Not very impressed with Bard code capabilities in my first experiments. I asked him a very basic Python task: to create a script that extracts data from a Postgres DB and save it in a csv file.

This is the result: https://pastebin.com/L3xsLBC2

Line 23 is totally wrong, it does not extract the column names. Only after pointing out the error multiple times he was able to correct it.

Re: Gemini AI

#328

Post by nojvek » Wed, Dec 06, 2023, 4:59 PM UTC

One of my biggest concerns with many of these benchmarks is that it’s really hard to tell if the test data has been part of the training data.

There are terabytes of data fed into the training models - entire corpus of internet, proprietary books and papers, and likely other locked Google docs that only Google has access to.

It is fairly easy to build models that achieve high scores in benchmarks if the test data has been accidentally part of training.

GPT-4 makes silly mistakes on math yet scores pretty high on GSM8k

Re: Gemini AI

#329

Post by sidcool » Wed, Dec 06, 2023, 5:01 PM UTC

This tweet by Sundar Pichai is quite astounding

https://x.com/sundarpichai/status/1732433036929589301?s=20

Re: Gemini AI

#330

Post by crazygringo » Wed, Dec 06, 2023, 5:01 PM UTC

behnamoh wrote: post #245

ilaksh wrote: post #58

Earlier quoted context omitted.

That's for Ultra right? Which is an amazing accomplishment, but it sounds like I won't be able to access it for months. If I'm lucky.

I hate this "tierification" of products into categories: normal, pro, max, ultra Apple does this and it's obvious that they do it to use the "decoy effect" when customers want to shop. Why purchase a measly regular iPhone when you can spend a little more and get the Pro version? But when it comes to AI, this tierification only leads to disappointment—everyone expects the best models from the FAANGO (including OpenAI)…

I don't understand -- these are all literally tied directly to performance.

They're tiers of computing power and memory. More performance costs more money to produce. The "nano" can fit on a phone, while the others can't.

Are you really objecting to the existence of different price/performance tiers...? Do you object to McDonald's selling 3 sizes of soft drink? There's nothing "decoy" about any of this.