Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLM 2 Technical Report [pdf]

ai.google

Post reply on HN
Print view View original

51–60 of 297 posts

Re: PaLM 2 Technical Report [pdf]

#51

Post by jiocrag »

ilikeatari wrote: post #9

So, I asked Bard if it's using PaLM 2 and it did confirm it. My initial results are super promising. Highly recommend checking it out again.

If Bard is using PaLM 2, Google is in serious trouble. Here's its offering for "the simplest PostgreSQL query to get month-over-month volume and percentage change." Note that no actual calculations take place and the query generates a syntax error because it references a phantom column. GPT 3.5 and 4 handle this with ease.

SELECT month, volume, percentage_change FROM ( SELECT date_trunc('month', created_at) AS month, SUM(quantity) AS volume FROM orders GROUP BY date_trunc('month', created_at) ) AS monthly_orders ORDER BY month;

Re: PaLM 2 Technical Report [pdf]

#52

Post by mrbungie »

sanxiyn wrote: post #49
jumpCastle wrote: post #37

Earlier quoted context omitted.

Optimazing for training could help distillation also.

I agree distillation is the wild card. The question is whether distillation works for LLM. I am not aware of any public report of successful distillation of LLM (I searched quite hard for this; if you know of any and can tell me I would be very grateful), and I interpreted it to mean that it doesn't work yet and negative results are not published due to publication bias.

This was published here in HN last week: https://news.ycombinator.com/item?id=35810663

Don't know if there any public technical reports by any of the big AI companies about this, as its pretty new.

Re: PaLM 2 Technical Report [pdf]

#53

Post by YetAnotherNick »

tempusalaria wrote: post #18
typon wrote: post #2

No comparisons against GPT-4 except on three benchmarks where PaLM 2 does better on two. Not sure why, but I expected better from Google.

Most of the GPT-4 benchmarks from their report were things like AP tests or leer code scores. Which aren’t benchmarks that can be compared by a different set of researchers as you don’t know the constituent parts of the test to run

GPT-4 report has MMLU score, which is believed to one of the most important metric for question answering task. GPT-4 MMLU score is slightly higher than PaLM 2(86 vs 81). Google didn't compare it in with PaLM 2 in this paper.

Re: PaLM 2 Technical Report [pdf]

#54

Post by execveat »

jacooper wrote: post #36
renewiltord wrote: post #31

Once they released its coding ability it became more useful. I use Bard less than ChatGPT still, but it is not useless since it has more modern information.

Is it better than bing or phind though? Why would I use it over bing?

In my experience Bing chat and phind are useless. But perplexity.ai and GPT4 are amazing. GPT-3.5 and Cloude-instant (available through poe.com) are cool as well, even though they got significantly dumbed down recently, presumably to lower the maintenance costs.

Re: PaLM 2 Technical Report [pdf]

#55

Post by minimaxir »

jxy wrote: post #23

So how do we actually try out the PaLM 2? The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API. How do I actually google the "PaLM API" for a way to test "PaLM 2"?

Google's docs on the APIs are up: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/...

The pricing is also now listed but free during the trial period, although it's annoyingly priced by character: https://cloud.google.com/vertex-ai/pricing#generative_ai_mod...

Assuming ChatGPT's tokens are the equivalent of 4 characters on average (a fair assumption), the pricing of PaLM's chat and embedding APIs are the same cost as OpenAI's equivalents.

Re: PaLM 2 Technical Report [pdf]

#56

Post by ntonozzi »

minimaxir wrote: post #55
jxy wrote: post #23

So how do we actually try out the PaLM 2? The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API. How do I actually google the "PaLM API" for a way to test "PaLM 2"?

Google's docs on the APIs are up: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/... The pricing is also now listed but free during the trial period, although it's annoyingly priced by character: https://cloud.google.com/vertex-ai/pricing#generative_ai_mod... Assuming ChatGPT's tokens are the equivalent of 4 characters on average (a fair assumption), the pricing of PaLM's chat and embedding APIs are the…

Why would that be annoying? It’s much easier to understand, predict and truncate appropriately than having to explain all of these different tokenization schemes to devs.

Re: PaLM 2 Technical Report [pdf]

#57

Post by [deleted] »

espadrine wrote: post #6

Surprisingly, their scaling law analysis still focuses on training FLOPs instead of training + inference FLOPs. That said, they do mention this: > The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute. [A] smaller but higher quality model significantly improves inference efficiency, reduces serving cost, and enables the model’s downstream…

[deleted]

Re: PaLM 2 Technical Report [pdf]

#58

Post by newhouseb »

jacooper wrote: post #34
jxy wrote: post #23

So how do we actually try out the PaLM 2? The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API. How do I actually google the "PaLM API" for a way to test "PaLM 2"?

It should be live on Bard.

But Google hasn't disclosed which version of Bard, right?

I pop into Bard every once in a while to test its performance, but I never know if I'm getting the best Google has or just what Google can tolerate running cost-wise publicly given they potentially have at least an order of magnitude (if not two, edit: 1.5) more users than OpenAI.

Re: PaLM 2 Technical Report [pdf]

#59

Post by rcoveson »

ntonozzi wrote: post #56
minimaxir wrote: post #55

Earlier quoted context omitted.

Google's docs on the APIs are up: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/... The pricing is also now listed but free during the trial period, although it's annoyingly priced by character: https://cloud.google.com/vertex-ai/pricing#generative_ai_mod... Assuming ChatGPT's tokens are the equivalent of 4 characters on average (a fair assumption), the pricing of PaLM's chat and embedding APIs are the…

Why would that be annoying? It’s much easier to understand, predict and truncate appropriately than having to explain all of these different tokenization schemes to devs.

Yeah, everybody agrees on what a character is, right? It's just {an ASCII byte|a UTF8 code unit|a UTF16 code unit|a Unicode code point|a Unicode grapheme}.

Re: PaLM 2 Technical Report [pdf]

#60

Post by ntonozzi »

rcoveson wrote: post #59
ntonozzi wrote: post #56

Earlier quoted context omitted.

Why would that be annoying? It’s much easier to understand, predict and truncate appropriately than having to explain all of these different tokenization schemes to devs.

Yeah, everybody agrees on what a character is, right? It's just {an ASCII byte|a UTF8 code unit|a UTF16 code unit|a Unicode code point|a Unicode grapheme}.

I’m not saying it’s easy but it’s much better than tokens IMO. I think bytes would be understandable too.
Post reply on HN