Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PaLM 2 Technical Report [pdf]

ai.google

Post reply on HN
Print view View original

141–150 of 297 posts

Re: PaLM 2 Technical Report [pdf]

#141

Post by knaik94 »

cdchn wrote: post #139
entropicdrifter wrote: post #123

Earlier quoted context omitted.

Agreed. I'm not entirely sure that the person you're replying to is not joking

I tried asking it "what is the difference between the palm language model and the bard language model?" and its reply started off "The main difference between the Palm language model and the Bard language model is the size of the dataset they are trained on. Palm is trained on a dataset of 400 billion parameters, while Bard is trained on a dataset of 540 billion parameters." Which to me is even more interesting that…

It makes up those numbers, I asked about the difference between the small and large PaLM 2 data set size, and it asserted the small model was trained on 540 billion and the large model was trained on 540 trillion. A different draft instead specified 1.4 trillion for the large.

Re: PaLM 2 Technical Report [pdf]

#143

Post by stygiansonic »

fzliu wrote: post #140

I don't understand how this can be considered a technical report. No information on model architecture, distributed training methodology, or optimizations. The "Training dataset" section is a pathetic 0.5 pages long. Come on, Google.

In that sense, it's very similar to the GPT-4 Technical Report.

The era of being "open" about LLMs or other "secret sauce" models in published papers may be over, since these things have become existential threats to companies.

Re: PaLM 2 Technical Report [pdf]

#144

Post by knaik94 »

It's very clear that the current Bard model is weaker than the largest PaLM 2 model. But for certain things, Bard seems worse than even the smallest model described. It's hard to say without someone doing a comprehensive benchmark, but the artifically limited context size makes testing useless for data.

The model was surprisingly confident when I tried to ask it about the relationship between better language comprehension and parameter size. The coherence displayed by the model, when it argued that a smaller model size will be capable of matching and surpassing competitive model performance, was a little jarring. Especially when, in the question right before, it said that the large PaLM 2 model has 540 trillion parameters.

Re: PaLM 2 Technical Report [pdf]

#145

Post by mk_stjames »

The paper begins with:

  Language modeling has long been an important research area since Shannon (1951) estimated the information in language with next word prediction.
Man, I wonder what Claude Shannon would think of all this if he were alive today...

Re: PaLM 2 Technical Report [pdf]

#146

Post by cdchn »

knaik94 wrote: post #141
cdchn wrote: post #139

Earlier quoted context omitted.

I tried asking it "what is the difference between the palm language model and the bard language model?" and its reply started off "The main difference between the Palm language model and the Bard language model is the size of the dataset they are trained on. Palm is trained on a dataset of 400 billion parameters, while Bard is trained on a dataset of 540 billion parameters." Which to me is even more interesting that…

It makes up those numbers, I asked about the difference between the small and large PaLM 2 data set size, and it asserted the small model was trained on 540 billion and the large model was trained on 540 trillion. A different draft instead specified 1.4 trillion for the large.

It even gave me a table with a whole bunch of differences. All thats made up?

Here is a table that summarizes the key differences between the two language models:

Feature Palm Bard Number of parameters 400 billion 540 billion Vocabulary size 137 billion words 1.5 trillion words

Re: PaLM 2 Technical Report [pdf]

#147

Post by haldujai »

stygiansonic wrote: post #143
fzliu wrote: post #140

I don't understand how this can be considered a technical report. No information on model architecture, distributed training methodology, or optimizations. The "Training dataset" section is a pathetic 0.5 pages long. Come on, Google.

In that sense, it's very similar to the GPT-4 Technical Report. The era of being "open" about LLMs or other "secret sauce" models in published papers may be over, since these things have become existential threats to companies.

I wonder how special these architectures are compared to what's published.

The "secret sauce" may just be getting 2 pages (~200) worth of engineers collaborating and either rolling out your own cloud service or spending $$$ at someone else's.

Also not sure how much it matters other than academic interest of course. Realistically, there's only 4-5 (US) companies with the human resources and capital to roll something similar to these models out for what is most likely a complete write-off?

They could claim whatever they wanted and it would be near impossible to validate.

Re: PaLM 2 Technical Report [pdf]

#148

Post by throwaway29303 »

jxy wrote: post #23

So how do we actually try out the PaLM 2? The links in their press release just link to their other press release, and if I google "PaLM API" it just gives me more press release, but I just couldn't find the actual document for their PaLM API. How do I actually google the "PaLM API" for a way to test "PaLM 2"?

https://developers.generativeai.google/guide

Re: PaLM 2 Technical Report [pdf]

#149

Post by EgoIncarnate »

knaik94 wrote: post #144

It's very clear that the current Bard model is weaker than the largest PaLM 2 model. But for certain things, Bard seems worse than even the smallest model described. It's hard to say without someone doing a comprehensive benchmark, but the artifically limited context size makes testing useless for data. The model was surprisingly confident when I tried to ask it about the relationship between better language comprehe…

The largest PaLM 2 model is smaller than 540 billion parameters of PaLM 1 (let alone 540 trillion!). From the PDF "The largest model in the PaLM 2 family, PaLM 2-L, is significantly smaller than the largest PaLM model but uses more training compute."

Re: PaLM 2 Technical Report [pdf]

#150

Post by chaxor »

gekoxyz wrote: post #84
jp42 wrote: post #62

personal experience - I'm using GPT4 for writing code especially in python. After using bard today, I feel bard is doing quite well considering its free. I will keep using it and if its keep doing well, I will cancel GPT4 $20/month subscription.

why don't you just use chatGPT? from what i know it's running GPT3.5 and it's not that different (at least in terms of code quality)

It's quite a vast difference between GPT-3.5 and GPT-4
Post reply on HN